Features Affecting Difficulty Parameters of Test Items Measuring the Skill of Reading Multiple Documents in English: Construct Definition and Representation

Abstract

The present study investigated factors that may affect the difficulty of test items that are intended to measure the skill of reading multiple documents. The participants, 151 undergraduates (28 women, 123 men) at several Japanese national universities, were asked to read pairs of passages in English that were on the same topic and answer 10 test items (also in English) about the content of the pairs of passages. The test items were presented in an extended matching format for each pair of passages, requiring the participants to classify the test sentences into one of the following 4 categories: only passage A said that, only passage B said that, both passages said that, and neither passage said that. The 2-parameter logistic constrained model (2PLCM) showed that the test items were more difficult when a test sentence targeted contents mentioned in both passages or in neither of them, compared to the other 2 types. No difference in item difficulty was observed between complementary and contradictory pairs of documents. The present results suggest that such factors should be taken into account when test items are being edited and reviewed.