Automated Essay Scoring With GPT-4 for a Local Placement Test: Investigating Prompting Strategies, Intra-Rater Reliability, and Alignment With Human Scores

Abstract

This study explores the potential of OpenAI's ChatGPT-4 (gpt-4-0613) as an automated essay scoring (AES) tool in a trial involving 300 essays from an American university's academic English program placement test. Three prompting strategies (minimal/detailed rubric, require/not require rationale, and with/without scoring examples) were tested for intra-rater reliability and agreement with human ratings and placements. The study also examined improvements through NLP-derived linguistic features and averaging multiple GPT-4 ratings. Results revealed promising indicators such as high intra-rater reliability, moderate positive correlations with human scores, and better placement agreement than the human average. Prompting strategies influenced scoring accuracy. A prompting strategy combining the provision of a detailed rubric, a request for a scoring rationale, and the provision of scoring examples demonstrated the best performance. Adding linguistic features and averaging multiple GPT-4 ratings were even more effective. The findings suggest that while relying entirely on GPT-4 for essay scoring might be premature, incorporating it as a complement to human ratings could be beneficial.