From helping hand to stumbling block: the ChatGPT paradox in competency experiment

Abstract

We ran a controlled laboratory experiment to examine whether ChatGPT’s aid can increase the participants’ performance in three different–reading and writing, mathematical problem-solving, and computational thinking–tasks. We find that the maths score significantly decreases with ChatGPT’s assistance. This result is mainly because the low-ability subjects couldn’t discern the hallucinated answers with the correct ones, and it contests the general idea that ChatGPT can complement the workers with less expertise.