Concurrent Validity and Inter-Rater Reliability Assessment of Two Mental Toughness Instruments in Firefighting: A Two-Wave Longitudinal Study

Abstract

Mental toughness (MT), a construct linked to high-stress performance, is predominantly studied via self-assessment in cross-sectional designs. In the firefighting context, where no dedicated MT instrument exists, the Sports Mental Toughness Questionnaire (SMTQ) and the Military Training Mental Toughness Inventory (MTMTI) have been used. However, their reliability and concurrent validity remain unexamined. This study employs a two-wave longitudinal design involving sixty-two male firefighters. Over two days, they completed the SMTQ, while colleagues and officers assessed their MT using the MTMTI. Analyses included concurrent validity and inter-rater reliability tests. Inter-rater reliability exhibited minimal agreement (Day 1: κ = 0.04, p = 0.172; Day 2: κ = 0.05, p = 0.063), low internal consistency (Day 1: α = 0.03, ω = 0.03; Day 2: α = 0.45, ω = 0.45), and weak inter-rater correlations (Day 1: ICC2k < 0.001, 95%CI [−0.53, 0.35]; Day 2: ICC2k = 0.13, 95%CI [−0.33, 0.43]). Concurrent validity showed limited correlations between self-assessed SMTQ scores and MTMTI ratings on both days (Day 1: r = −0.09, p = 0.5; Day 2: r = 0.1, p = 0.5). These findings may underscore the necessity to develop firefighting-specific MT tools, recognizing the unique demands and nuances of this high-stress profession.