Automatic Scoring System for English Writing Based on Natural Language Processing: Assessment of Accuracy and Educational Effect

Abstract

This study investigates the effectiveness and educational impact of a novel NLP-based English writing auto-scoring system. Utilizing advanced machine learning techniques, including BERT and Graph Neural Networks, the system demonstrates high consistency with human raters (Quadratic Weighted Kappa of 0.92) across multiple dimensions of writing quality. A longitudinal study involving 500 students over a 16-week semester revealed significant improvements in writing abilities, with the most substantial gains observed in grammar and mechanics (28.5% increase) and organization and structure (23.7% increase). Through comprehensive system evaluation using multiple metrics, including Adjacent Agreement Rate and Root Mean Square Error, our system consistently outperformed existing baseline approaches, including commercial off-the-shelf solutions. The implementation of our system significantly enhanced teacher efficiency, reducing essay grading time by 62% and increasing time for individualized feedback by 45%. The system’s architecture integrates cutting-edge NLP technologies with a user-friendly interface, facilitating real-time feedback and adaptive assessment capabilities. Our evaluation framework encompasses both technical accuracy and educational effectiveness, addressing a critical gap in current literature. While the system shows limitations in assessing highly creative writing and faces potential risks of student gaming, its overall impact on writing instruction and assessment is overwhelmingly positive. The study demonstrates that NLP-based auto-scoring systems can effectively scale writing assessment, provide timely feedback, and potentially democratize access to high-quality writing instruction. These findings suggest a path toward more efficient, personalized, and equitable writing education.