Understanding patterns of loneliness in older long-term care users using natural language processing with free text case notes

Abstract

Loneliness and social isolation are distressing for individuals and predictors of mortality, yet data on their impact on publicly funded long-term care is limited. Using recent advances in natural language processing (NLP), we analysed pseudonymised administrative records containing 1.1 million free-text case notes about 3,046 older adults recorded in a London council between 2008 and 2020. We applied three NLP methods— document-term matrices, pre-trained embeddings, and transformer-based models—to identify loneliness or social isolation. The best-performing model, a bidirectional transformer, achieved an F1 score of 0.92 on a test set of unseen sentences. Using this model, we generated predictions for the full dataset and assessed construct validity through comparison with survey data and the literature. Our measure is associated with expected characteristics, such as living alone and impaired memory, and is a strong predictor of social inclusion services. Approximately 43% of individuals had a sentence indicating loneliness or isolation in their case notes at their initial care assessment, comparable to survey-based estimates. Unlike surveys, our indicator is linked to other administrative data, enabling development of models of service use with loneliness or isolation as independent variables. An open-source version of the model is available in a GitHub repository.