AI detectors are poor western blot classifiers: a study of accuracy and predictive values

Abstract

The recent rise of generative artificial intelligence (AI) capable of creating scientific images presents a challenge in the fight against academic fraud. This study evaluates the efficacy of three free web-based AI detectors in identifying AI-generated images of western blots, which is a very common technique in biology. We tested these detectors on AI-generated western blot images (n = 48, created using ChatGPT 4) and on authentic western blots (n = 48, from articles published before the rise of generative AI). Each detector returned a very different sensitivity (Is It AI?: 0.9583; Hive Moderation: 0.1875; and Illuminarty: 0.7083) and specificity (Is It AI?: 0.5417; Hive Moderation: 0.8750; and Illuminarty: 0.4167), and the predicted positive predictive value (PPV) for each was low. This suggests significant challenges in confidently determining image authenticity based solely on the current free AI detectors. Reducing the size of western blots reduced the sensitivity, increased the specificity, and did not markedly affect the accuracy of the three detectors, and only slightly improved the PPV of one detector (Is It AI?). These findings highlight the risks of relying on generic, freely available detectors that lack sufficient reliability, and demonstrate the urgent need for more robust detectors that are specifically trained on scientific contents such as western blot images.