Being reliable: Issues in determining the reliability and making sense of observations of adults with congenital deafblindness?

Abstract

Background Most research into interactions with people who are congenitally deafblind involves observational data. In order for practitioners and researchers to have confidence in the findings of observational studies, researchers need to demonstrate that the processes employed are replicable and trustworthy. This paper draws on data from an observational study of adults with congenital deafblindness to illustrate issues in determining inter-rater reliability, and interpreting observational data. Method Data from 34 10-min observations of adults with congenital deafblindness and their interactions with support staff were assessed for inter-rater reliability using percentage agreement calculated in three different ways and Cohen's κ. Results Large variation resulted from the different ways in which inter-rater reliability was calculated, largely due to high levels of non-occurrence of many behaviours in the coding tool used. Conclusion This study highlights the need to exercise caution when ascertaining the reliability of observational studies and demonstrates the value in using multiple methods for calculating inter-rater reliability. The paper concludes with an examination of the potential merits of using consensus coding in observational studies of interactions with people with congenital deafblindness or profound intellectual and multiple disabilities. © 2011 Blackwell Publishing Ltd.