Inter-rater reliability of physical abuse determinations in young children with fractures

Abstract

As there is no “gold standard” in determining whether a fracture is caused by accident or abuse, agreement among medical providers is paramount. Using abstracted medical record data from 551 children <36 months of age presenting to a pediatric emergency department, we examined the extent of agreement between specialists who evaluate children with fractures for suspected abuse. To simulate clinical scenarios, two pediatric orthopaedists and two child abuse pediatricians (CAPs) reviewed the full abstraction and imaging, whereas the two pediatric radiologists reviewed a brief history and imaging. Each physician independently rated each case using a 7-point ordinal scale designed to distinguish accidental from abusive injuries. For any discrepancy in independent ratings, the two specialists discussed the case and came to a joint rating. We analyzed 3 types of agreement: (1) within specialties using independent ratings, (2) between specialties using joint ratings, and (3) between clinicians (orthopaedists and CAPs) with more versus less experience. Agreement between pairs of raters was assessed using Cohen's weighted kappa. Orthopaedists (κ = 0.78) and CAPs (κ = 0.67) had substantial within-specialty agreement, while radiologists (κ = 0.53) had moderate agreement. Orthopaedists and CAPs had almost perfect between-specialty agreement (κ = 0.81), while agreement was much lower for orthopaedists and radiologists (κ = 0.37) and CAPs and radiologists (κ = 0.42). More-experienced clinicians had substantial between-specialty agreement (κ = 0.80) versus less-experienced clinicians who had moderate agreement (κ = 0.60). These findings suggest the level of clinical detail a physician receives and his/her experience in the field has an impact on the level of agreement when evaluating fractures in young children.