Retracting ChatGPT: completeness and relevance of academic references

Abstract

This two-part study describes a learning exercise including a video and a worksheet designed to raise students’ awareness of the need to evaluate the completeness of references generated by ChatGPT. The first part of the study assesses the completeness and relevance of academic references generated by ChatGPT using four prompts and three versions of ChatGPT. Content analysis using a priori coding revealed that most of the references generated are hallucinations or only partially complete. The results from the first part of the study were utilized in the second part of the study where 66 students evaluated the completeness of a sample of the references that had been generated and evaluated in the first part of the study. Although most students participating in the case study correctly answered questions about the content of the videos, only some students were able to correctly evaluate the completeness of the generated references.