You cannot improve what you do not measure: A triangulation study of software security metrics

Abstract

When organizations invest in security, they need to monitor if their security program is effective and helps them remediate vulnerabilities. For this purpose, many organizations collect security metrics. In this paper, we investigate the current state-of-the-art and state-of-practice of security metrics used to measure security across all phases of software development lifecycle (SDLC). The study focused on gaining multiple perspectives on software security measurement. To this end, we performed a triangulation study that compared security metrics proposed in the academic literature, metrics mentioned in grey literature aimed at software practitioners, and metrics elicited in a focus group workshop with secure software engineering experts.Our study reports two critical insights. First, our results reveal a significant discrepancy in the utilization of metrics across the different SDLC stages. While the academic literature proposes a comprehensive spectrum, encompassing metrics for both early and late SDLC phases, industry predominantly focuses on the later SDLC stages. This highlights an industry-wide tendency to prioritize security measurement later in the software development process, potentially overlooking early-stage concerns.Second, our study sheds light on the practitioners' dissatisfaction with the current security metrics. This dissatisfaction highlights the industry's need for more nuanced and effective metrics that can offer both quantitative and qualitative insights to assess security of a software development program.