The Use of Stocking-Lord and Haebara Methods in Horizontal Equating: A Case of Indonesian Madrasah Competence Assessment

Abstract

Indonesian Madrasah Competence Assessment (AKMI) is a national assessment implemented each year held by the Ministry of Religious Affairs. One uniqueness of the AKMI is the use of different tests every year. AKMI focuses on capturing the learning development in Madrasah by comparing the test scores of the current year with those of the previous year. An equating process is crucial for valid results when comparing scores. Therefore, this research aims to (a) equate the scientific literacy assessment tools at AKMI in 2022 with 2023 and (b) evaluate the business process of developing AKMI scientific literacy instruments (along with the MSAT design), which has implications for the equating process. This study adopted a Non-Equivalent Anchor Test (NEAT) design because the two test sets were parallel years, and the participants were from a diverse population. The data is from the AKMI Science Literacy of the Ministry of Religious Affairs, with 303,987 participants in 2022 and 342,987 in 2023 from the Islamic elementary school level. There were 674 scientific literacy instrument items in 2022 and 1,392 items in 2023, with 90 items used as anchor items. There are three stages of analysis: pre-equalisation, equalisation calibration, and post-equalisation analysis. The results show differences in item parameter estimation results between 2022 and 2023, where 2022 has a higher level of item difficulty. Furthermore, the Stocking-Lord and Haebara methods were effective and produced estimates with minimal differences in the equating process. In addition, the anchor items used as the basis for the equating do not represent the items as a whole in the item pool. These findings indicate the need for firm, careful standardisation based on psychometric principles of the process at AKMI, from developing items to assembling items, testing, determining anchor items, and assembling items in the MSAT application.