Format Effects of Empirically Derived Multiple-Choice Versus Free-Response Instruments When Assessing Graphing Abilities

Abstract

Prior graphing research has demonstrated that clinical interviews and free-response instruments produce very different results than multiple-choice instruments, indicating potential validity problems when using multiple-choice instruments to assess graphing skills (Berg & Smith in Science Education, 78(6), 527–554, 1994). Extending this inquiry, we studied whether empirically derived, participant-generated graphs used as choices on the multiple-choice graphing instrument produced results that corresponded to participants’ responses on free-response instruments. The 5 – 8 choices on the multiple-choice instrument came from graphs drawn by 770 participants from prior research on graphing (Berg, 1989; Berg & Phillips in Journal of Research in Science Teaching, 31(4), 323–344, 1994; Berg & Smith in Science Education, 78(6), 527–554, 1994). Statistical analysis of the 736 7th – 12th grade participants indicate that the empirically derived multiple-choice format still produced significantly more “picture-of-the-event” responses than did the free-response format for all three graphing questions. For two of the questions, participants who drew graphs on the free-response instruments produced significantly more correct responses than those who answered multiple-choice items. In addition, participants having “low classroom performance” were affected more significantly and negatively by the multiple-choice format than participants having “medium” or “high classroom performance.” In some cases, prior research indicating the prevalence of “picture-of-the-event” and graphing treatment effects may be spurious results, a product of the multiple-choice item format and not a valid measure of graphing abilities. We also examined how including a picture of the scenario on the instrument versus only a written description affected responses and whether asking participants to add marker points to their constructed or chosen graph would overcome the short-circuited thinking that multiple-choice items seem to produce.