Natural Language Processing of Electronic Patient Records to Predict Psychiatric Inpatients at Risk of Early Readmission to Hospital Using Predictive Models Derived Through Machine Learning

Abstract

Aims. The authors designed a simulation training programme for foundation doctors beginning psychiatry placements across a large mental health trust. The simulation training aimed to improve the confidence, competence, and well-being of foundation doctors through exposing them to realistic psychiatry scenarios and teaching clinical skills in a safe environment. Methods. Four clinical scenarios were filmed with a 360-degree camera, professional actress, and doctors working in psychiatry. The scenarios depicted the journey of a patient being admitted onto a psychiatry ward from the community. Various clinical skills were embedded into the videos including psychiatric history taking, risk assessment, managing acute distress, managing comorbid physical and mental health problems, using the Mental Health Act, and teamwork with colleagues. All videos were delivered to learners using simulation with head-mounted-displays (HMDs). Each video lasted 6-8 minutes and was accompanied by pre-briefing and de-briefing with experienced psychiatrists for a further 15-20 minutes. Participants rated their confidence regarding several skills in psychiatry on Likert scales from 1 to 5 immediately before and after the session. Wilcoxon signed rank tests were conducted to detect statistically significant differences in learner's median confidence ratings before and after the training. Free-text questions explored trainee's most and least favourite aspects of the simulation. A survey also was distributed to learners 2-months after the training to assess how it had influenced their clinical practice. Results. 20 foundation doctors completed the training and provided feedback. Following the simulation training, there were statistically significant improvements in foundation doctor's confidence in: completing psychiatric assessments (p < 0.01), managing physical health problems in psychiatry (p < 0.05), managing acute distress (p < 0.01), reporting information to senior colleagues (p < 0.05), and containing anxiety when communicating with patients (p < 0.05). Trainees highlighted the debriefing, group discussions, and "interactive" simulation videos as the most useful aspects of the training. Some trainees enjoyed viewing the 360-degree videos, whilst others found the HMDs difficult to use. Of the 8 trainees who completed feedback 2 months after the training, 7 (87.5%) felt that it had helped them in their current roles. All trainees agreed (37.5%) or strongly agreed (62.5%) that the simulation scenarios were closely aligned to real-life clinical encounters. Conclusion. Simulation training in psychiatry using 360-degree videos and HMDs is generally well-received amongst foundation doctors. Embedding simulation training into placement induction can improve the confidence and skills of junior doctors starting psychiatry placements. Aims. Psychiatric readmissions cause a burden on the healthcare system, incur a monetary cost and cause additional distress to acutely unwell patients. This project explores the use of the free-text of electronic patient records to predict inpatients in psychiatric hospitals at risk of readmission using predictive models generated by machine learning. Methods. Free-text was extracted from the electronic patient records of patients admitted to hospitals in Birmingham and Solihull Mental Health Foundation Trust (BSMHFT) during the five years 2015-2019 inclusive. The anonymised records were obtained via the CRIS (Clinical Record Interactive Search) database. A total of 17208 records were extracted. The free-text entered by clinicians during an admission was extracted and processed using techniques of natural language processing to generate input vectors suitable to be used with machine learning algorithms. tf-idf (term frequency-inverse document frequency) vectors were used. A selection of algorithms were used to train predictive models. Two-thirds of the records were used as training data with the remainder as test data. Baseline model performance was assessed and then best-performing candidates underwent hyperparameter optimisation using five-fold cross-validation to improve performance. Bayesian optimisation was used to automate hyperparameter tuning during cross-validation. Hyperparameters were optimised on the log loss function. As the dataset was imbalanced with negative instances outnumbering positive instances to a significant degree, various techniques such as random undersampling of negative instances in the training data were used to deal with class imbalance throughout this process. Following cross-validation, the best-performing models underwent performance analysis. Models were used to make predictions on the test data. Performance was assessed using F1-measures, precision-recall curves and the average precision metric (equivalent to area under the precision-recall curve). These metrics were chosen due to their suitability in assessing models trained on imbalanced datasets. Results. The best F1 score obtained was 0.233 using a Random Forest model trained using unigram tf-idf vectors of 500 token dimension. The best average precision obtained was 0.157 using a Support Vector Machine trained using unigram tf-idf vectors of 2000 token dimension. Both the above results required the use of random oversam-pling of positive instances to improve performance on the imbal-anced dataset. Conclusion. The performance indicates that the models generated are unlikely to have significant practical utility. Nevertheless, this exploratory project has produced a processed dataset with knowledge about its characteristics. This could be used for the further development of models using more complex techniques such as language modelling using neural networks. S6 Rapid-Fire Presentation