A Distributed Federated Reinforcement Learning Approach for Scheduling User-Side Flexibility Resources in Virtual Power Plants

Abstract

The high penetration of renewables increases the volatility of the net load, revealing flexibility shortages in distribution networks. This paper proposes a coordination framework for multiple virtual power plants (VPPs) that aggregates distributed flexibility resources (e.g., electric vehicles, battery energy storage systems and shiftable loads) to enhance the adaptability of the system. The flexibility supply–demand imbalances is first quantified by using unified margin and risk indices, then a cost-flexibility co-optimisation model is formulated to align local dispatch with system requirements. A distributed federated reinforcement learning (FRL) algorithm with doubly stochastic aggregation is developed for decentralised, privacy-preserving and communication-efficient training. Unlike centralised approaches, the proposed FRL algorithm enables distributed agents to collaboratively learn optimal scheduling strategies through local interactions and neighbour-based model updates, improving scalability, robustness and adaptability. The results of the case studies demonstrate rapid convergence, a significant improvement in peak-period flexibility insufficiency and reduced operational costs, thereby validating the effectiveness of our proposed approach in improving future grid flexibility and overall economic performance.