Word Predictability as a Measure of Second Language Proficiency

Abstract

This study introduces predictabilityBERT, a novel metric for assessing second language (L2) proficiency based on the predictability of word choices in learner language production. Using BERT (Devlin et al., 2019), we calculated the conditional probability of each word in a text given its surrounding context. We evaluated predictabilityBERT on two datasets: the Lexical Proficiency Corpus (N = 480), containing analytic ratings of lexical proficiency, and TOEFL 11 (N = 11,000; Blanchard et al., 2013), containing standardized language proficiency scores. Results show that predictabilityBERT correlated with ratings of collocation accuracy (r =.80) and lexical proficiency (r =.73). In a multilevel model, predictabilityBERT was the strongest predictor of language proficiency compared to conventional measures of lexical and phraseological sophistication, explaining 59% of variance in TOEFL scores. These findings suggest high-proficiency L2 learners make more predictable word choices, supporting usage-based theories emphasizing the role of statistical learning in L2 development.