Interrater reliability of the BelRAI Social Supplement in Flanders, Belgium: Simultaneous rating of community-dwelling adults with care needs during COVID-19

Abstract

Background: The BelRAI Screener is a short-form assessment consolidating internationally validated interRAI items focusing on physical and psychological aspects of functioning and problems with activities of daily living. It was fully implemented in the Flemish home care setting as of June 2021. In a biopsychosocial model for developing a personalized and effective care plan social and contextual aspects are considered equally important to biomedical ones. Thus, a social supplement to the BelRAI Screener was collaboratively developed with stakeholders and tested to gather additional information on the social context of community-dwelling adults with care needs. Objective and methods: To examine the interrater reliability of the BelRAI Social Supplement in Flanders, Belgium, an observational study was conducted using a convenience sample. The method of simultaneous rating was used due to strict COVID-19 guidelines at the time and to minimize assessment burden. Fifty two community-dwelling adults requesting home care support were simultaneously assessed by two independent assessors during home visits. Interrater reliability was tested on all 80 items of the BelRAI Social Supplement using observed agreement, kappa coefficients, and intraclass correlation coefficients. Results: The kappa mean (0.74) and median (0.79) values for nominal items, show substantial agreement, while the kappa mean and median values for ordinal items were 0.81 and 0.90, which correspond to almost perfect agreement. Following the traditional cut-off points for the interpretation of the kappa statistic, reliability was almost perfect (κ > 0.81) for 49% of all items, substantial (0.60 < κ ≤ 0.80) for 33%, moderate (0.40 < κ ≤ 0.60) for 8%, and poor (κ < 0.40) for 10%. The majority of items with poor kappa value, showed a high observed agreement, reflecting homogeneity of the sample rather than poor agreement. Conclusion: The strength of kappa agreement for the items in this version of the BelRAI Social Supplement is generally substantial to almost perfect, with high proportions of observed agreement. COVID-19 restrictions had a large impact on the planning and execution of the home visits. A final optimization of the instrument and accompanying manual according to the findings will result in an improved version ready for nation-wide implementation.