Improving and Analyzing Open-Ended Survey Responses: A Case Study Linking Psychological Theories and Analysis Approaches for Text Data

Abstract

Responses to open-ended items (OEI) in surveys in evaluation research are valuable but often short or even missing. We investigate how response behavior can be altered by means of survey design. In a survey on tutoring courses, 3,183 students responded to OEI. Participants were randomly assigned to conditions with or without a motivational cue and with OEI positioned at the beginning or the end of the survey, respectively. We simultaneously inspected response rates and answer lengths using Bayesian multilevel hurdle models. Positioning at the beginning enhanced response rates but reduced answer lengths. The motivational cue raised response rates only at the end of the survey. We also applied two different topic modeling approaches. Resulting topics cover different aspects of teacher behavior and positioning slightly altered the content of the responses. Based on motivational and cognitive theories, we address both the design of questionnaires and analysis approaches to OEI responses.