A Reinforcement Learning Model for Optimal Treatment Strategies in Intensive Care: Assessment of the Role of Cardiorespiratory Features

Abstract

Goal: The purpose of this study is to evaluate the importance of cardiorespiratory variables within a Reinforcement Learning (RL) recommendation system aimed at establishing optimal strategies for drug treatment of septic patients in the intensive care unit (ICU). Methods: We developed a RL framework in order to establish drug administration strategies for septic patients by exclusively using a set of cardiorespiratory variables. We then compared this model with other equivalent models trained with a wider set of clinical features, as well as several models with similar dimensionality. We selected patients meeting the Sepsis-3 criteria from the Multi-parameter Intelligent Monitoring in Intensive Care (MIMIC III) database, resulting in a total of 20,496 ICU admissions. Data were normalized and then organized into discrete time-series. A Markov Decision Process (MDP) was built on the discrete time-series using the k-means++ clustering algorithm to define the states and a grid of 25 possible actions composed by a combination of vasopressors and intravenous fluid dosages. A policy iteration algorithm was applied to solve the MDP and obtain the optimal AI policy. The policy performance was then evaluated with the use of the weighted importance sampling estimator (WIS estimator). The process was repeated for each set of variables and compared with a set of baseline benchmark policies. Results: The model trained with cardiorespiratory variables outperformed all other models considered, resulting in a 95% confidence lower bound score of 97.48. This finding highlights the importance of cardiovascular variables in the clinical RL recommendation system. Conclusions: We established an efficient RL framework for sepsis treatment in the ICU and demonstrated that consideration of cardiorespiratory variables provides critical information in devising optimal policies. Given the potentially continuous availability of cardiorespiratory features extracted from bedside physiological waveform monitoring, the proposed framework paves the way for a real time recommendation system for sepsis treatment.