splithalf: robust estimates of split half reliability

Abstract

License Authors of papers retain copyright and release the work under a Creative Commons Attribution 4.0 International License (CC BY 4.0). splithalf: robust estimates of split half reliability The R package splithalf provides tools to estimate the internal consistency reliability of cog-nitive measures. In particular, the tools were developed for application to tasks that use difference scores as the main outcome measure, for instance the Stroop score or dot-probe attention bias index (average RT in incongruent trials minus average RT in congruent trials). The methods in splithalf are built around split half reliability estimation. To increase the robustness of these estimates, the package implements a permutation approach that takes a large number of random (without replacement) split halves of the data. For each permutation the correlation between halves is calculated, with the Spearman-Brown correction applied (Spearman, 1904). This process generates a distribution of reliability estimates from which we can extract and plot summary statistics (e.g. average and 95% HDI). Statement of need While many cognitive tasks yield highly robust effects, the same task may not yield reliable individual differences (Hedge et al., 2018). As these measures are used in questions of individual differences researchers need to have some psychometric information for the outcome measures. Recently, it was proposed that psychological science should set a standard expectation for the reporting of reliability information for cognitive and behavioural measures (Parsons et al., 2019). splithalf was developed to support this proposition by providing a tool to easily extract internal consistency reliability estimates from behavioural measures. Usage The splithalf function estimates reliability for a variety of scoring types, including average scores, difference scores, and difference-of-difference scores. The function is also extendable to other scoring calculations, e.g. signal detection. Reliability can be estimated for response time and accuracy rate outcomes. A plotting option is also provided for users to examine the distribution of reliability estimates. Users can perform a reliability multiverse analysis to examine the stability of reliability estimates , given a rage of data processing decisions (Parsons, 2020). By passing a list of data processing decisions (e.g. maximum and minimum RT cutoffs) and a splithalf object into the splithalf.multiverse function. testretest.multiverse can be used to explore a similar multiverse of data processing decisions on test retest reliability. Multiverse output can be plotted with plot_multiverse.