Detecting Plagiarism in the Age of Generative AI: An Exploratory Experiment

Abstract

Generative AI agents such as ChatGPT have created renewed concerns about the adverse effects of technology on students’ learning through receiving unpermitted aid in their coursework. We conducted an exploratory experiment involving a typical college course assignment to detect and compare genuine student responses with responses generated by ChatGPT. Using a text classification scheme that we devised, we showed that student responses are fairly accurately distinguishable from AI’s, not only when AI uses its general knowledge to answer questions, but also when it is prompted to use the same material used by students. In addition, we identified elements of authorship style, primarily related to formality, that can help humans set students’ and AI’s work apart. We also explored the strategies that students use and the depth at which they alter AI-generated content to make it “their own”. We found the alterations to be mostly moderate, and the modified text remained mostly detectable by our classif ication scheme. Overall, our results offer a transparent machine learning model for detecting AI-generated text, stylistic cues that can help humans detect such text, as well as insights into students’ strategies when borrowing content from AI agents.