Same Question But Different Answer: Experimental Evidence on Questionnaire Design's Impact on Poverty Measured by Proxies

Abstract

Based on a randomized survey experiment that was implemented in Malawi, the study finds that observationally-equivalent, as well as same, households answer the same questions differently depending on whether they are interviewed with a short questionnaire or its longer counterpart. Statistically significant differences in reporting emerge across all topics and question types. In proxy-based poverty measurement, these reporting differences lead to significantly different predicted poverty rates and Gini coefficients. The difference in poverty predictions ranges from 3 to 7 percentage points, depending on the model specification. A prediction model based only on the proxies that are elicited prior to the variation in questionnaire design yields identical poverty predictions irrespective of the short-versus-long questionnaire treatment. The results are relevant for estimating trends with questionnaires exhibiting inter-temporal variation in design, impact evaluations administering questionnaires of different length and complexity to treatment and control samples, and development programs utilizing proxy-means tests for targeting.