Renewable Energy Bidding Strategies Using Multiagent Q-Learning in Double-Sided Auctions

Abstract

Owing to energy liberalization and increasing penetration of renewables, renewable energy trading among suppliers and users has gained much attention and created a new market. This article investigates a double-auction scheme operated by an aggregator with limited supervision for energy trading. To ensure beneficial bidding for renewable generators and end users (EUs) considered as agents, a multiagent Q-learning (MAQL) based bidding strategy is developed to maximize their cumulative reward. Each agent first provides their information about renewable supply or demand to an aggregator who will then return the information about aggregate supply and demand. Without knowing the business model of the aggregator, the agents use Q-tables to estimate the expected cumulative reward and determine their bidding prices accordingly. Finally, the aggregator coordinates energy trading between agents who will then update their Q-tables on the basis of the amount of power bought or sold at the prices they bid. The proposed approach can avoid some unnecessary or unrealistic assumptions generally made by model-based approaches, such as the assumption on the knowledge of others' bidding profiles or the assumption of an oligopoly; it can consider the influence of bidding strategies on the market, which cannot be properly addressed by a conventional proportional allocation mechanism. A numerical analysis using real-world data and considering a profit maximization model for the aggregator shows that the proposed approach outperformed comparable methods in terms of profits of renewable generators and energy satisfaction level of EUs: iterative double auction by approximately 29%, heuristic bidding by 39.9%, random bidding by 38.1%, NSGA-II-based multiobjective approach by 62%, and MOEA/D-based multiobjective approach by 83.1% on average.