Development and Evaluation of a Component Level Implementation Fidelity Rating System for the GenerationPMTO Intervention

Abstract

Empirically determining the components of evidence-based interventions contributing to positive change is a crucial, yet understudied area of research. In support of this aim, we describe the development and evaluation of an observational rating system for measuring fidelity to specific components of the evidence-based GenerationPMTO parenting intervention. A five-step process was employed to systematically develop the rating system, which included consultation with the intervention developer and input from additional GenerationPMTO experts. The rating system was then tested using 247 h of video data from 184 parenting group intervention sessions. Study findings support the psychometric properties of the new measure with regard to item performance, reliability (i.e., inter-rater reliability of items, dimensionality of components, internal consistency of component scales), and validity (i.e., content validity, convergent validity, discriminant validity, and predictive validity of the component scales) for seven of the eight scales evaluated. The seven components include clear directions, skill encouragement, emotion regulation, limit setting, effective communication, problem solving, and monitoring. Data did not support the psychometric properties of the positive involvement scale. Overall, the ability to assess component-specific fidelity allows for a more nuanced examination of change processes, with meaningful implications for research and practice.