Outcomes of endovascular treatments of aneurysms: Observer variability and implications for interpreting case series and planning randomized trials

Abstract

BACKGROUND AND PURPOSE: Angiographic results are commonly used as a surrogate marker of success of coiling of intracranial aneurysms. Inter- and intraobserver agreement in judging angiographic results remain poorly characterized. Our goal was to offer such an evaluation of a grading scale commonly used to evaluate results of endovascular treatment of aneurysms. MATERIALS AND METHODS: A portfolio of 90 angiographic images from 45 patients selected from the core lab data base of a randomized trial was sent to 12 observers on 2 occasions more than 3 months apart. The variability of a 3-value grading scale used to score angiographic results and of a final judgment regarding the presence of a recurrence was studied using κ statistics. RESULTS: Ten participants responded once and 6 responded twice. Agreement was poor to moderate (κ = 0.28-0.5) for senior and junior observers judging angiographic results immediately or 12-18 months after treatment. Agreement reached a reassuring "substantial" (κ = 0.62) level, with a dichotomous presence-absence of a major recurrence, and intraobserver agreement was better in experienced core lab assessors. CONCLUSIONS: There is an important variability in the assessment of angiographic outcomes of endovascular treatments, rendering comparisons between publications risky, if not invalid. A simple dichotomous judgment can be used as a surrogate outcome in randomized trials designed to assess the value of new endovascular devices.