Toward Model-Assisted Safe Reinforcement Learning for Data Center Cooling Control: A Lyapunov-based Approach

Abstract

This paper considers intelligent data center cooling control via the Deep Reinforcement Learning (DRL) approach to improve data center sustainability. Existing DRL-based controllers are trained with a simplified data hall thermodynamic model which assumes uniform room temperature distribution. This assumption is not valid for a real-world data center with highly nonuniform temperature distribution. Furthermore, most of them cannot guarantee thermal safety during the DRL learning process. To bridge these gaps, we propose LyaSafe, a model-assisted safe DRL approach for data center cooling control. To address the safety evaluation issue, we develop a coupled model that combines a differentiable surrogate data hall thermodynamics model with the energy model. It can simulate both data hall temperature distribution and the facility energy consumption. To address safe learning, we introduce a novel constrained Markov Decision Process (CMDP) formulation for data center cooling control by considering the Rack Cooling Index (RCI), the best-practice metric for evaluating compliance with ASHRAE data center thermal guidelines. The objective is to minimize data center carbon footprints while regulating the RCI within a threshold. We first derive the safety set based on the concept of the virtual queue and Lyapunov stability theory. Next, we rectify unsafe actions from the DRL agent by projecting them to the safety set. We evaluate LyaSafe in a data center hosting 20 racks and 299 servers. Evaluation results show that LyaSafe can ensure strict safety during the DRL learning while achieving up to 50 metric tons of annual carbon emission savings using Singapore's statistics. Moreover, we conduct root cause analysis for the savings, revealing the importance of joint control of the data hall and the chiller plant.