Coh-Metrix Model-Based Automatic Assessment of Interpreting Quality

Abstract

The present study applied the computational tool Coh-Metrix to analyze linguistic and discoursal features of the transcribed interpreting texts from the All China Interpreting Contest (ACIC) in order to predict quality assessments provided by the judges in the ACIC. We find that a stepwise linear regression model with four entries/variables (i.e., word count, lexical diversity, hypernymy of verbs, and frequency of first person singular) could predict 60% of the variance in the human scoring. These results highlight the importance of information completeness in the ACIC, as word count was highly correlated with the human scoring. In addition, the interpreted texts produced by the better performing contestants/interpreters were associated with an increasing level of language sophistication. That is, they tended to use words of higher precision, less frequency, and higher diversity. Based on these results, we argue that even without recourse to diagnostic information related to pronunciation, intonation, rhythm, and speed of delivery, automatically computed analytic indices such as word count, word diversity, and hypernymy of verbs could predict a significant amount of variance in human scoring of interpreting quality. These encouraging, albeit preliminary, findings will prompt us to further explore Coh-Metrix model-based automatic assessment of interpreting quality and also help us gain an enhanced understanding of the “quality” construct perceived by the judges in the ACIC.