Large Language Models Improve Scene-Invariant Detection of Behavior of Risk in Dementia Residential Care Across Multiple Surveillance Camera Views

Abstract

Behavioral and psychological symptoms of dementia pose challenges to the safety and well-being of individuals in residential care. The integration of video surveillance in common areas of these settings presents a valuable opportunity for developing automated deep learning methods capable of identifying such behavior of risk. By issuing real-time alerts, these methods can support timely staff intervention and reduce the likelihood of incidents escalating. However, a persistent limitation is the considerable drop in performance when these methods are deployed in environments unseen during training. To address this issue, we propose an unsupervised scene-invariant fusion-based deep learning network. It combines language model-based captioning and scoring with video anomaly detection scoring to improve the generalization performance for unseen camera scenes. The video anomaly detection scoring uses a depth-weighted spatio-temporal autoencoder to reduce false positives, and the caption-based scoring uses a large language model to generate anomaly scores from captions of video frames. The study uses video data collected from nine individuals with dementia, recorded via three distinct hallway-mounted cameras in a dementia unit. The performance was investigated in both the same camera and cross-camera settings, where the proposed method performed consistently better than the existing methods. The proposed approach obtained the best area under receiver operating characteristic curve performance of 0.855, 0.84 and 0.805 for the three cameras. This work motivates further research to develop cross-camera behavior of risk detection systems for people with dementia in care environments.