On Using Simulations to Inform Decision Making During Instrument Development

Abstract

Although content validity—how well each item of an instrument represents the construct being measured—is foundational in the development of an instrument, statistical validity is also important to the decisions that are made based on the instrument. The primary purpose of this study is to demonstrate how simulation studies can be used to assist the decision making of researchers who are developing or updating instruments. A hypothetical research study is presented in which the researcher needs to make choices that will guide his or her development of a universal behavior screener. In the study, he or she wishes to make choices regarding the number of items to include, the pilot sample size, average difficulty of the items, and the amount of information provided by the instrument at different cut scores. Simulation is then used to create data sets with varying levels of each of these aspects, and decisions are subsequently made regarding the levels that should be applied in the actual study. Rationale for these decisions as well as implications for practice are included.