Using analytic rating scales to assess English–Chinese bi-directional interpreting: A longitudinal Rasch analysis of scale utility and rater behaviour

Abstract

Descriptor-based analytic rating scales have been used increasingly to assess interpreting quality. However, little empirical evidence is available to support unequivocally the effectiveness of rating scales and rater reliability. This longitudinal study therefore attempts to provide insight into scale utility and rater behaviour in the assessment of English–Chinese interpreting performance, using multifaceted Rasch measurement. Specifically, the study focuses on criterion or scale difficulty, scale effectiveness, rater severity or leniency and rater self-consistency between English–Chinese interpreting and over three time points. The research results are discussed, highlighting the utility of analytic rating scales and the variability of rater behaviour in assessing interpreting. The results also have implications for developing reliable, valid and practical instruments to assess the quality of interpreting.