From Reports to Actions: Bridging the Customer Usability Gap in Penetration Testing

Abstract

Penetration testing reports play a significant role in helping organizations identify and mitigate security vulnerabilities. The report effectiveness relies on the extent to which customers can translate the findings into actionable decisions. Our study investigates usability gaps in penetration testing reports from a customer-centric perspective, focusing on the challenges organizations face in understanding, prioritizing, and acting on the provided insights. Within the study, we demonstrated a penetration testing scenario together to IT professionals scaled from technical to managerial. We provided them with a selected reported vulnerability finding from the scenario. By conducting surveys and focus groups, we aimed to identify common gaps. Based on data from 25 focus group participants, we conducted a thematic analysis, identifying 8 themes with 29 findings categorized as possible improvements, gaps, and general perceptions, highlighting weaknesses in PT reports. The results highlight the necessity of aligning report content with the needs of specific target audiences. Key findings reveal gaps in defining the scope, rules of engagement, and methodology before conducting PT, as well as inadequacies in describing findings within reports. Additionally, our research underscores the importance of incorporating positive findings and enhancing security recommendations by avoiding generic mitigations, providing multiple mitigation options, assessing their impact, and specifying preferred solutions when applicable. The study explores how these gaps can affect decision-making processes, risk mitigation efforts, and overall cybersecurity outcomes. By highlighting these issues, our research sheds light on the need for improved usability in penetration testing deliverables to better serve customer needs and enhance cybersecurity outcomes.