Development of a Multidimensional Fidelity Observation Tool for Social Competence Intervention

Abstract

There has been a growing emphasis on identifying evidence-based practices across fields. Successful evidence-based practices depend on multiple constructs including implementation fidelity (Carroll et al., 2007). Governing research institutes (e.g., NIH, IES) have called for implementation fidelity in both ideal and routine contexts. High achievement of fidelity increases successful replication of interventions beyond original research conditions, thus encouraging effective dissemination .(Carroll et al., 2007). Despite growing emphasis on employment of fidelity measures to enhance intervention implementation, little empirical research exists on such constructs (Sanetti & Kratochwill, 2009). While obtaining fidelity bolsters the value of research, limitations to current fidelity measures exist. Traditional methods often obtain fidelity through the employment of checklists or implementer ratings (Eames et al., 2007). Thus, use of traditional methods may lack ability to draw conclusions of effectiveness without presence of quantitative components. Additionally, traditional measures may lack the following multi-dimensional components: adherence to content and process of intervention, dosage, participant responsiveness, and program differentiation (Carroll et al., 2007). Cost of inadequate fidelity could lead to not only rejection of powerful programs but also acceptance of weaker programs (Borelli et al., 2005). If the field is not able to replicate effective programs due to lack of fidelity, then dissemination of best practices cannot occur. For these reasons we investigated the call for high-level fidelity through the employment of observation fidelity tool investigating the school-based social competence intervention (SCI) program. This study is part of a larger RCT examining the efficacy of the SCI program across one Midwest state. To ensure effective replication across implementers and strength of this program compared to no treatment, we measured implementation fidelity using a tool developed for the study. The purpose of this paper is to present psychometric properties of a quantitative fidelity tool. Subjects: Twelve middle schools were recruited to be included in the first year of implementation of the SCI program, as part of an IES efficacy and replication grant. Six schools were randomly selected to receive training and support for the SCI program. The remaining six schools served as the comparative group. Implementers included special education teachers and speech language pathologists. Procedures: Procedures for this quantitative fidelity observation tool occurred in two phases: development and implementation. To develop the fidelity measure we identified and defined core components based on theoretical constructs of social competence (Stichter et al., 2010). Additionally, we used two types of interval recording methods and ratings to capture multi-dimensional components as proposed by Carroll et al. (2007). Next, we trained a coding team until 85% interobserver reliability was met. For the implementation phase, two coders recorded every SCI sessions either in vivo or via videotapes. Results and Conclusion: Reports of internal validity and reliability of this fidelity tool are presented. Traditional methods lack the breadth to draw conclusions regarding effectiveness. Thus, anticipated conclusions for implementers' adherence to the SCI program, (content and delivery skills, dosage, and student responsiveness) inform the strength of this treatment program, and dissemination, training, and support efforts (Borelli et al., 2005). (PsycEXTRA Database Record (c) 2013 APA, all rights reserved)