Telephone Sample Surveys: Dearly Beloved or Nearly Departed? Trends in Survey Errors in the Era of Declining Response Rates

Abstract

Telephone surveys have become much maligned in the past few years, considering recent failures to correctly predict elections worldwide, response rates declining into the single digits, and the rise of low-cost, nonprobabilistic alternatives. Yet there is no study assessing the degree to which data attained via modern-day telephone interviewing has or has not significantly declined in terms of data quality. Utilizing an elemental approach, we evaluate the bias of various cross-tabulations of core demographics from a collection of surveys collected over the past two decades. Results indicate that (1) there has been a modest increase in bias over the past two decades but a downward trend in the past five years; (2) the share of cell phone interviews in samples has a significant impact on the bias; (3) traditional weighting largely mitigates the linear trend in bias; and (4), once weighted, telephone samples are nearly on par in data quality to higher response rate unweighted in-person data. Implications for the "fit for purpose"of telephone data and its general role in the future of survey research are discussed given our findings.