Measuring cognitive reflection without maths: Development and validation of the verbal cognitive reflection test

Abstract

The cognitive reflection test (CRT) became popular for its impressive power to predict how well people reason and make decisions. Despite the popularity of the CRT, a major issue complicates its interpretation: The numerical nature of the CRT confounds reflection ability with mathematical ability. We have addressed this issue by developing the verbal CRT (CRT-V), a novel 10-item measure of cognitive reflection (https://osf.io/xehbv/), using nonmathematical problems with good statistical and psychometric properties and with low familiarity. First, we selected suitable items with relatively low familiarity and optimal difficulty as identified in two different populations (Studies 1 and 2) and with high content validity as judged by an expert panel (Study 3). Second, we demonstrated good criterion and construct validity for the test in different populations with a wide range of variables (Studies 4–6, 8) and a good internal consistency (Studies 4–8) and test–retest reliability (Study 7). The CRT-V was less associated with math anxiety, objective and subjective numeracy than the original CRT, and it was test equivalent across gender, age groups and administration setting. In contrast with the original CRT (Hedge's g = 0.29, 95% confidence interval [CI] [0.17, 0.40]), the CRT-V showed no gender differences (Hedge's g = −0.06, 95% CI [−0.18, 0.06]). The CRT-V can complement existing, numerical, tests of cognitive reflection.