New test-taking patterns and their effect on language test validity

Abstract

Educators, especially test creators, are concerned with the construct validity of their tests. Blame has generally been attached to test-wiseness strategies as one source of error in measurement. There is little evidence of any relation between test-taking strategies in general and test validity; thus, it is not known how strategies can affect test validity. In addition, the literature does not report studies that have measured the effectiveness of the test-taking strategies used by learners. In this study, we propose to detail 12 test-taking patterns observed in data gathered from 42 EFL learners on two different vocabulary achievement tests, using think-aloud protocols and interviews. Although there were logical patterns, we also discovered some unusual patterns. In some cases, students managed to arrive at the right answers that were words they did not know using test-wiseness strategies, whereas other students got items wrong although they knew the correct word because of test-unwiseness strategies. In addition to showing how strategies affect test validity, the degree of effectiveness of some strategies was measured. Some factors were addressed that played a role in the effectiveness of certain strategies as well as factors causing the more unusual patterns.