Chasing the Unicorn? The Feasibility of Automatic Assessment of Interpreting Fluency

Abstract

This chapter examines the feasibility of using Praat for automatic assessment of interpreting fluency (i.e., speed and breakdown fluency). A total of 140 audio recordings were collected in an English-to-Chinese consecutive interpreting exam. Two raters assessed the fluency of the interpreting performance using a four-point scale and their mean scores were used as the perceived fluency scores. Two versions of the audio files, unedited and edited (with inter-segmental silences and noises being removed), were subjected to Praat analysis in two conditions (intensity-adjusted versus intensity-unadjusted). In the intensity-adjusted condition, the intensity threshold was set differently for individual recordings, depending on the maximum intensity and the 99% quantile intensity. In the intensity-unadjusted condition, the threshold was set uniformly for all the audio files. The correlations between Speed Fluency, Breakdown Fluency, and Perceived Fluency were examined across the four conditions (i.e., edited and intensity-adjusted, edited and intensity-unadjusted, unedited and intensity-adjusted, unedited and intensity-unadjusted). Statistical analyses showed that: (a) removing inter-segmental silences and noises slightly improved the correlation coefficients; (b) adjusting the intensity threshold significantly improved the correlation coefficients; (c) the silent pause threshold of 0.25 second produced the best correlation coefficients; and (d) Mean Silence Duration had the strongest correlation with judged fluency. Based on these findings, a flowchart is designed to guide teachers’ decision-making in fluency assessment, ranging from pre-processing of audio files, to configuration of parameters in Praat, and finally to relating acoustic measures to assessment goals.