Who's on First? Gender Differences in Performance on the SAT ® Test on Critical Reading Items With Sports and Science Content

Abstract

This study gathered quantitative and qualitative evidence concerning gender differences in performance by using critical reading material on the SAT ® test with sports and science content. The fundamental research questions guiding the study were: If sports and science are to be included in a skills test, what kinds of material are appropriate for fair and valid assessments? Does item type matter with regard to differential item functioning (DIF)? What factors can provide explanations for the different levels of difficulty, discrimination, and DIF exhibited? Do differences in students' interest in and familiarity with the material affect their performance? From data gathered during the two phases of our research, it appears that most sports and science material of the sort used in this study should be permissible in a skills test. Particularly in the context of a long passage, in which most information needed to answer an item is provided, very little material appears to be too “specialized.” From participants' statements during cognitive interviews, it seems that interest in and/or familiarity with a subject has little impact on performance—despite previous research to the contrary. On the basis of our observations, we recommend the following: (a) use more than one type of statistic to evaluate differential item functioning; (b) consider the amount of context provided with an item rather than just the particular content of an item; and (c) when possible, use data rather than assumptions about gender bias to make decisions about test content that may generate differential item functioning. Report Number: ETS RR‐16–26