Are We Skillful or Just Lucky? Interpreting the Possible Histories of Vulnerability Disclosures

Abstract

Coordinated Vulnerability Disclosure (CVD) stands as a consensus response to the persistent fact of vulnerable software, yet few performance indicators have been proposed to measure its efficacy at the broadest scales. In this article, we seek to fill that gap. We begin by deriving a model of all possible CVD histories from first principles, organizing those histories into a partial ordering based on a set of desired criteria. We then compute a baseline expectation for the frequency of each desired criteria and propose a new set of performance indicators to measure the efficacy of CVD practices based on the differentiation of skill and luck in observation data. As a proof of concept, we apply these indicators to a variety of longitudinal observations of CVD practice and find evidence of significant skill to be prevalent. We conclude with reflections on how this model and its accompanying performance indicators could be used by various stakeholders (vendors, system owners, coordinators, and governments) to interpret the quality of their CVD practices.