Using paradata to explore item level response times in surveys

Abstract

We analyse item level keystroke data from cycle 6 of the National Survey of Family Growth, which is a survey on fertility and related topics that is conducted in the USA. The National Survey of Family Growth is conducted among both males and females by using computer-assisted personal visit interviews and an audio computer-assisted self-interviewing component for the most sensitive topics. Our analyses focus on the time taken to answer a question as a function of item level characteristics, respondent characteristics and interviewer characteristics. Using multilevel models, we explore how these factors influence response times. Our exploratory study suggests that factors at all three levels (item, respondent and interviewer) influence response times. These results demonstrate that question features that explain variation in response times can be automatically derived from standard computer-assisted personal interviewing paradata. The effects of respondent characteristics that we observe are in line with prior findings from more controlled studies conducted in supervised telephone facilities. Some demographic characteristics of interviewers contributed to the variation in response times, though they failed to explain large portions of the between-interviewer variance. © 2012 Royal Statistical Society.