Exploring Market Designs for Enhanced Flexibility Procurement with Deep Reinforcement Learning

Abstract

The growing share of renewable energy in shortterm European electricity markets has significantly increased congestion management costs and demands. Therefore, current market design is not optional to keep congestion costs low. A proper market would incentivize the integration of flexibilities to boost competition and lower costs, while mitigating risks of manipulation. However, assessing behavioral impacts is challenging due to increasingly interconnected market structures. Studies modeling more than two markets often overlook the strategic opportunities that emerge from these interactions, focusing instead on large-scale dynamics. To capture the detailed impact of bidding strategies, we use reinforcement learning to explore multi-market strategies. By progressively training a Deep Reinforcement Learning (DRL) agent as a market participant - from replicating established behaviors to mastering intricate multimarket interactions - we employ Domain-Informed Curriculum Learning (DomCL), a structured approach that systematically guides learning through staged complexity. We validate our approach against established two-market studies, then evaluate it in two progressively complex four-market case studies spanning a 6-bus network, including historical data. Results show that our DRL-based method improves performance while uncovering challenges that arise as strategic opportunities expand, offering a structured approach for multi-market design analysis.