Assessing spoken language ability: A many-facet Rasch analysis

Abstract

Assessing speaking in a useful way has been attended with some concerns, such as scoring subjectivity and test bias. Most often, scoring the speaking performance might result in some unfairness that could possibly emanate from these two issues. This, in turn, could harm the life of test-takers and many other stakeholders. This article investigated the assessment of speaking among learners of English in an ESP program in an EFL context. To this end, 213 test-takers were assessed on their speaking ability by 12 raters using a five-rubric scale (task achievement, fluency, grammar, vocabulary and phonology). Each candidate was assessed by two raters, totaling six pairs of two raters each). The speaking exam included four parts only of which two parts were graded, thus excluding the opening and closing sections. All the exam questions were pre-formulated and teachers were instructed to stick to the frame of the exam. The results of the study showed that generally the test-takers’ speaking ability was scored more leniently than harshly and that raters were biased towards the speaking rubrics, which indicated a fuzzy idea about such rubrics. Assuredly, the different statistical tests of the FACETS showed that the speaking exam was neither valid nor reliable. The study had relevant implications for rater training in how to score speaking in an objective way and for recommending writing a list of test specifications (specs) to design useful and fair speaking exams in similar-related contexts.