BENFORD'S LAW AS AN INDICATOR OF SURVEY RELIABILITY—CAN WE TRUST OUR DATA?

Abstract

This paper analyzes how closely different income measures conform to Benford's law, a mathematical predictor of probable first digit distribution across many sets of numbers. Because Benford's law can be used to test data set reliability, we use a Benford analysis to assess the quality of six widely used survey data sets. Our findings indicate that although income generally obeys Benford's law, almost all the data sets show substantial discrepancies from it, which we interpret as a strong indicator of reliability issues in the survey data. This result is confirmed by a simulation, which demonstrates that household level income data do not manifest the same poor performance as individual level data. This finding implies that researchers should focus on household level characteristics whenever possible to reduce observation errors.