Automated Assessment of Creativity in Multilingual Narratives

Abstract

Researchers and educators interested in creative writing need a reliable and efficient tool to score the creativity of narratives, such as short stories. Typically, human raters manually assess narrative creativity, but such subjective scoring is limited by labor costs and rater disagreement. Large language models (LLMs) have shown remarkable success on creativity tasks, yet they have not been applied to scoring narratives, including multilingual stories. In the present study, we aimed to test whether narrative originality—a component of creativity— could be automatically scored by LLMs, further evaluating whether a single LLM could predict human originality ratings across multiple languages.We trained three different LLMs to predict the originality of short stories written in 11 languages. Our first monolingual model, trained only on English stories, robustly predicted human originality ratings (r=.81). This same model—trained and tested on multilingual stories translated into English—strongly predicted originality ratings of multilingual narratives (r≥.73). Finally, a multilingual model trained on the same stories, in their original language, reliably predicted human originality scores across all languages (r ≥.72).We thus demonstrate that LLMs can successfully score narrative creativity in 11 different languages, surpassing the performance of the best previous automated scoring techniques (e.g., semantic distance). This work represents the first effective, accessible, and reliable solution for the automated scoring of creativity in multilingual narratives.