Testing the Reliability and Validity of Net-HARMS: A New Systems-Based Risk Assessment Method in HFE

Abstract

There is growing interest in the use of systems-based risk assessment (RA) methods in human factors and ergonomics (HFE). Despite this, there has been a lack of formal reliability and validity testing undertaken to determine whether systems-based RA methods have the capacity to reliably and accurately identify potential risks within complex systems. The purpose of this study was to test the intra-rater reliability (within subject stability) and criterion-related validity (‘gold standard’ performance) of the Networked Hazard Analysis and Risk Management System (Net-HARMS). Net-HARMS is a new and innovative systems-based RA method that supports analysts with the identification of emergent risks. Emergent risks represent new risks that are created when risks from across a complex system interact with one another. Reliability and validity measures for Net-HARMS were obtained using the Signal Detection Theory (SDT) paradigm. Matthews Correlation Coefficient (MCC) was used to analyze the complete SDT data to measure the strength of the correlation between risks. Findings indicate a weak to moderate level of reliability and validity for Net-HARMS based on the MCC score. The results suggest that there is merit to the continued use of Net-HARMS following a series of methodological recommendations that aim to enhance the reliability and validity of future applications.