Evaluating the effectiveness of the training program on direct and semi-direct oral proficiency assessment: A case of multifaceted Rasch analysis

Abstract

An Oral Proficiency Interview (OPI) may be evaluated either during the interview procedure (direct-method) or from a tape-made oral interaction (semi-direct method). Such variety in methods of assessment can influence test takers’ scores extensively. However, it is not conclusive whether such differences are due to test format or something else. Besides, most previous studies have investigated the application of multifaceted Rasch measurement (MFRM) on only one or two facets and few studies have used a pre- and post-training design. 20 English as a foreign language (EFL) teachers rated the oral performances produced by 200 test takers before and after a training program. The tasks were implemented via two methods of task delivery, direct and semi-direct. The findings indicated that test takers found semi-direct oral tests harder and more stressful than direct oral tests since they are more difficult due to having more complicated linguistic and communicative features and contained more lexically complex structures. Therefore, direct oral tests are more appropriate for low-level test takers, and semi-direct tests for higher ability ones. Data analyses demonstrated no significant difference between the ratings of direct and semi-direct oral assessment by the raters. Consequently, semi-direct oral tests can be regarded as a reliable substitute for direct oral tests. Finally, the outcomes showed that although training can result in higher levels of interrater consistency and reduced levels of severity/leniency, biasedness and inconsistency, it cannot turn rater into duplicates of one another.