Synthetic Social Engineering Scenario Generation Using LLMs for Awareness-Based Attack Resilience

Abstract

Social engineering is found in a strong majority of cyberattacks today, as it is a powerful manipulation tactic that does not require the technical skills of hacking. Calculated social engineers utilize simple communication to deceive and exploit their victims, all by capitalizing on the vulnerabilities of human nature: trust and fear. When successful, this inconspicuous technique can lead to millions of dollars in losses. Social engineering is not a one-dimensional technique; criminals often leverage a combination of strategies to craft a robust yet subtle attack. In addition, offenders are continually evolving their methods in efforts to surpass preventive measures. A common utility to defend against social engineering attacks is detection-based software. Security awareness, however, is a valuable approach that is often eclipsed by automated tech solutions. Awareness establishes a strong first line of defense against these ever-changing attacks. This study utilizes four data-supplemented large language models to generate custom social engineering scenarios with the goal of supporting strong example-driven security awareness programs. The performances of BERT, T5, GPT-3.5, and Llama 3.1 are comparatively analyzed, with Llama 3.1 producing the highest quality scenarios based on multiple metrics, including LLM-as-a-judge. Through chain-of-thought prompting, the Llama 3.1: Interactive version is capable of generating superiorly realistic, detailed, and specific attack scenarios, which are invaluable to improving hands-on curriculums and security exercises.