Student test-taking effort in low-stakes assessments: evidence from the English version of the PISA 2015 science test

Abstract

Background: The idea of using low-stakes assessment results is often mentioned when designing educational system reforms. However, when tests have no consequences for the students, test takers may not make enough effort when completing the test, and their lack of engagement may negatively affect the validity of the conclusions of the studies that use such tests. This article presents analyses of student test-taking engagement in a low-stakes international large-scale assessment, the Programme for International Student Assessment (PISA), and (i) quantifies test-taking effort, (ii) predicts effort by means of expectancy-value proxies and (iii) investigates the relationship between effort and science proficiency. Methods: Students’ response times on the science items of PISA 2015 were used to derive an index of test-taking effort. Data from six English-speaking countries that administered the computer-based version of the test, were selected. The response time for each item was modelled by means of a two-class finite mixture model, and students’ probabilities of being classified as effortful were combined to derive a global index of effort. Results: Our findings showed that students’ effort decreased towards the end of the testing sessions. The variance of examinees’ test-taking effort was not substantially explained by the expectancy-value variables. Test-taking effort had a strong relationship with science achievement, with the correlation increasing to more than 0.5 towards the end of the test. Moreover, an important part of the relationship between test-taking effort and achievement is not related to other student characteristics, such as gender, socio-economic and cultural status, attitude towards school or attitude towards science. Conclusions: This study shows that students put different amounts of effort into test taking, especially towards the end of the assessment, and suggests a possible underestimation of related student achievement that may affect the interpretation of test scores.