A within-sample investigation of test-retest reliability in choice experiment surveys with real economic incentives

Abstract

In this paper, we investigate the level of agreement between respondents' choices in identical choice sets in a test-retest choice experiment for a market good with real economic incentives, thus investigating whether the incentivised CE method can be reliable and stable over time. Besides comparing choices, we also test for differences in preferences and error variance when a sample of respondents is given the exact same questionnaire twice, with a time lag of 2 weeks in between. Finally, we examine potential reasons and covariates explaining the level of agreement in choices across the 2 weeks. Across four different tests, we find very good agreement between the two choice experiments - both with respect to overall choices and with respect to preferences. Furthermore, error variances do not differ significantly between the two surveys. The results also show that the larger the utility difference in a choice task, the larger the probability that the respondent will choose the same alternative in the retest. Moreover, the results show that the longer time respondents take to answer the 12 choice sets in the retest, the lower the probability that the respondent will choose the same alternatives in the retest as they did in the test.