Evaluating the measurement error of interviewer observed paradata

Abstract

As survey researchers have begun exploiting paradata-for example, for the correction of nonresponse bias-the quality of these data has come into question. Inaccurate information is likely to affect the resulting statistics and conclusions drawn from such data. This paper focuses on one type of paradata, observations made by interviewers during the data-collection process, and assesses the quality of these observations by examining their measurement error properties. The analysis uses the UK Census Nonresponse Link Study, which links interviewers' observations collected on six major UK surveys with Census data. Comparing five interviewer observations with self-reports from the Census, the accuracy of the observations for both respondents and nonrespondents to the surveys is evaluated. A multilevel modeling approach is used to explore under which conditions the interviewers' observations match the reports on the Census forms, accounting for the clustering of sample members within interviewers and areas. The analysis finds that the overall percent agreement between the observations and the Census is generally high, ranging from 87 to 98 percent. The type of housing structure and the final result code are significantly associated with measurement error. For four of the five observations, there is evidence that the interviewer significantly influences the level of measurement error, even after controlling for household, interviewer, and area characteristics. The results presented here will inform future analyses assessing the quality of interviewers' observations. © 2013 The Author.