Is there a method to our madness? The impact of data collection methodology on organizational survey results

Abstract

Results from 3 organizational survey efforts were used to explore the impact of data collection method on 5 outcomes: response preference, item completion rate, use of don't know responses, item mean and variability. The first study compared 977 online with 711 opscan responses in a U.S. government agency. The second study compared a Time 2 administration of 893 online with 732 opscan responses. The third study examined 4,654 automated phone (IVR) with 3,587 paper responses in a global pharmaceuticals firm. Although method utilization varied considerably by country, the more technological approaches were preferred in the U.S., and younger employees were significantly more likely to respond online. Overall, survey method accounted for a relatively small percentage of unique variance in the data (0% to 4%). Practitioners may be better off choosing an administration method based on factors such as cultural fit and ease of implementation rather than issues of data quality.