Interobserver agreement among sleep scorers from different centers in a large dataset

Abstract

Study Objectives: To evaluate epoch by epoch agreement in sleep stage assignment between scorers from different laboratories. Design: N/A. Methods: 62 NPSGs were selected for analysis from 3 sleep centers (38 diagnostic studies for sleep disordered breathing [SDB], 10 studies during CPAP titration, and 14 studies in subjects with no sleep related complaints or sleep pathology). The sleep recording montage consisted of at least 2 EEG leads, left and right EOG and a submental EMG. Scoring was performed manually by 5 experienced sleep technologists. No scorer had knowledge of any other scorers' results. Agreement was tabulated both for sleep stage distribution and on an epoch by epoch basis for the entire data set and the normal and SDB subsets. Measurements and Results: The mean epoch by epoch agreement between scorers for all records was 73% (range 67-82%). Agreements were higher in the normal subset (mean 76%, range 65-85%) than in the SDB subset (mean 71%, range 65-78%). There was significant variability in agreement between records and between pairs of scorers. Overall, 75% of epochs had at least 4 of the 5 scorers in agreement on the sleep stage and 96% of epochs had agreement of at least 3 of the 5 scorers. Conclusions: The level of agreement in sleep stage assignment varies between scorers, by diagnosis, and by record. The level of agreement between laboratories is lower than what can be maintained between scorers within the same laboratory. This warrants caution when comparing data scored in separate laboratories. The lower agreement in SDB patients supports the generally held view that sleep fragmentation makes application of the R and K rules less reliable.