Dissecting Reliability and Validity Evidence of Subjective Creativity Assessment: A Literature Review

Abstract

The importance of creativity for learning, an equitable education, and a competitive nation warrants a broader and deeper understanding of this topic, including how creativity is assessed. This review focuses on subjective creativity assessment, a popular assessment approach that uses judges’ subjective definitions of creativity, and examines its reliability and validity evidence collected from 84 empirical studies under the theoretical frameworks of the 2014 Standards for Educational and Psychological Testing and rater-mediated assessments. The main findings include: 1) The reviewed studies vary across domains, characteristics of subjects/objects and raters, and rating instructions and scales; 2) The major reliability evidence was provided by Cronbach’s alpha and correlations of rating scores, and the major validity evidence came from the evidence based on relationships with other variables through the use of correlations; 3) Cronbach’s alpha values differed through an interaction between domains and judges’ expertise level, and correlations of rating scores differed by domain and judges’ expertise level; 4) There was strong convergent validity evidence between creativity and novelty but a weak discriminant validity evidence between creativity and technical goodness and liking. These findings suggest that the subjective creativity assessment approach shows a good level of reliability and validity but has some degrees of unreliability and invalidity that need to be addressed with good research practices and more advanced measurement theories and methods.