How audits fail according to accident investigations: A counterfactual logic analysis

Abstract

Despite the reliance on safety auditing within organizations, comparatively limited research has studied the performance of safety auditing. When an investigation laments the “lack of audit quality” following an accident, what is meant by this statement? What contrasts a “good quality” audit from a “poor quality” audit? This study examined counterfactual logics (statements about alternative realities that did not occur but “could have” according to investigators) within 44 major accident reports to assess how audits are supposed to function and how they fall short of the ideal model. The content analysis yielded nine counterfactual auditing failures grouped into four categories. Contrary to the “ideal” model, audits (a) failed to facilitate an accurate understanding of threats by misinterpreting their saliency, (b) failed to facilitate timely action against threats by inadequately addressing the deterioration of known issues, (c) failed to facilitate effective management of issues, leading to confusion around the purpose and scope of audits, and d) failed to facilitate sufficient focus on threats by lacking focus on critical hazards and focusing on paperwork over operational issues or “failing silently” by missing threats while simultaneously praising performance. Practitioners should critically evaluate audits against these criteria and ensure audits effectively identify early warning signs.