Comparison of Evaluation Metrics for Short Story Generation

Abstract

This study aimed to analyze the correlation among different automatic evaluation metrics for short story generation. In the study, texts were generated from short stories using different language models: the N-gram model, the Continuous Bag-of-Word (CBOW) model, the Gated recurrent unit (GRU) model and the Generative Pre-trained Transformer 2 (GPT-2) model. All models were trained on short Aesop's fables. The quality of the generated text was measured with various metrics: Perplexity, BLEU score, the number of grammatical errors, Self-BLEU score, ROUGE score, BERTScore, and Word Mover's Distance (WMD). The resulting correlation analysis of the evaluation metrics revealed four groups of correlated metrics. Firstly, perplexity and grammatical errors were moderately correlated. Secondly, BLEU, ROUGE and BERTScore were highly correlated. Next, WMD was negatively correlated with BLEU, ROUGE and BERTScore. On the other hand, Self-BLEU, which measures text diversity within the model, did not correlate with the other metrics. In conclusion, to evaluate text generation, a combination of various metrics should be used to measure different aspects of the generated text.