Scrutinizing the basis of originality in divergent thinking tests: On the measurement precision of response propensity estimates

Abstract

Background: The originality of divergent thinking (DT) production is one of the most critical indicators of creative potential. It is commonly scored using the statistical infrequency of responses relative to all responses provided in a given sample. Aims: Response frequency estimates vary in terms of measurement precision. This issue has been widely overlooked and is addressed in the current study. Sample and method: Secondary data analysis of 202 participants was performed. A total of 900 uniquely identified responses were generated on three DT tasks and subjected to a 1-parameter logistic model with a response as the unit of measurement which allowed for the calculation of response-level conditional reliability (and marginal reliability as an overall summary of measurement precision). Results: Marginal reliability of response propensity estimates ranged from.62 to.67 across the DT tasks. Unique responses in the sample (the basis for the classic uniqueness scoring) displayed the lowest conditional reliability (across tasks: ≈.50). Reliability increased nonlinearly as a function of both the frequency of occurrence predicted by the model (conditional reliability) and sample size (conditional and marginal reliability). Conclusions: This study indicates that the common practice of frequency-based originality scoring with typical sample sizes (e.g., N = 100 to N = 200) yields unacceptable levels of measurement precision (i.e., in particular for highly original responses). We further offer recommendations to mitigate the lack of measurement precision of frequency-based originality scores for DT research.