LLM-driven bot infiltration: protecting web surveys through prompt injections

Abstract

Cost- and time-efficient web surveys potentially help covering the increasing survey data demand. However, since web surveys face low response rates, researchers consider social media platforms for recruitment. Although these platforms provide targeting tools, data quality and integrity might be threatened by bots. Established bot detections are not reliable when it comes to LLM-driven bots linked to Large Language Models (LLMs). We therefore investigate whether and to what extent prompt injections help detecting LLM-driven bots in web surveys. We instructed two LLM-driven bots with cumulative skillsets (LLM and LLM+) to respond to an open-ended question. This question included no injection, a jailbreaking injection, or a prompt leaking injection. Our results indicate that both bots react differently to prompt injections. While the less sophisticated LLM bot falls for the jailbreaking injection, the more sophisticated LLM+ bot falls for the prompt leaking injection. This indicates that prompt injections must be tailored to bot sophistication.