Optimal reinforcement learning control of a district cooling system based on compound secondary sampling under real-time electricity prices

Abstract

To achieve carbon neutrality in the power system, renewable energies (RENs), such as wind and solar power, are being rapidly installed. However, due to their strong intermittency and high uncertainty, balancing power supply and demand is becoming increasingly difficult. Time-of-use and real-time pricing (RTP) are crucial methods to ensure effective consumption of RENs, encouraging flexible resources on the demand side to work with RENs through demand-side response. Air conditioning loads represent a significant portion of urban loads, with >50% of the peak load in China. They can use the building’s thermal inertia to provide regulation services, a current area of focus for demand-response research. To adapt to the RTP mechanism in the future electricity market, this study explores demand-side optimization control technology for an ice storage system in a large-scale district cooling system (DCS). An optimal control method for an ice storage system is proposed, based on compound second-sampling reinforcement learning (RL), which can effectively manage uncertainties from users’ cooling demands and real-time market prices. First, a Markov decision process (MDP) is constructed to address the operational control issue of the DCS. Second, a model-free RL algorithm is used to solve the MDP. Third, the compound second-sampling mechanism is proposed to improve training efficiency and convergence performance by combining immediate return and temporal-difference error to overcome the issue of low learning efficiency caused by uniform random sampling in the traditional RL algorithm. Finally, the experimental results confirm the effectiveness of the proposed method.