AI-generated language in introductory physics lab reports in the first year of ChatGPT

Abstract

This article investigates 797 introductory college and university physics lab reports to find the prevalence and character of AI-generated language in them after the release of ChatGPT. The lab reports were inspected manually and using two AI detection services, GPTZero and Turnitin. In the year after the release of ChatGPT, generative AI language was found in 7.7% of lab reports. Students who used generative AI tended to use it multiple times. Six qualitative features of generative AI language were identified and given labels; verbose generality, fake concern, absence of numbers, suspicious formatting, significant ‘waffling’, and unrelated example. The AI detection services were found to be useful in detecting AI-generated language as they can highlight suspected sentences. Their AI detection scores and verdicts generally correlated with the presence of manually found generative AI language. The presence of false positives is demonstrated, as is the difficulty the technical writing nature of physics lab reports give the AI detection services.