Evaluating Transformer Models and Human Behaviors on Chinese Character Naming

Abstract

Neural network models have been proposed to explain the grapheme-phoneme mapping pro¬cess in humans for many alphabet languages. These models not only successfully learned the correspondence of the letter strings and their pronunciation, but also captured human behav¬ior in nonce word naming tasks. How would the neural models perform for a non-alphabet language (e.g., Chinese) unknown character task? How well would the model capture hu¬man behavior? In this study, we first collect human speakers’ answers on unknown Char¬acter naming tasks and then evaluate a set of transformer models by comparing their performance with human behaviors on an un¬known Chinese character naming task. We found that the models and humans behaved very similarly, that they had similar accuracy distribution for each character, and had a sub¬stantial overlap in answers. In addition, the models’ answers are highly correlated with humans’ answers. These results suggested that the transformer models can capture humans’ character naming behavior well.