Factored MDPs for optimal prosumer decision-making in continuous state spaces

Abstract

The economic profitability of Smart Grid prosumers (i.e., producers that are simultaneously consumers) depends on their tackling of the decision-making problem they face when selling and buying energy. In previous work, we had modelled this problem compactly as a factored Markov Decision Process, capturing the main aspects of the business decisions of a prosumer corresponding to a community microgrid of any size. Though that work had employed an exact value iteration algorithm to obtain a near-optimal solution over discrete state spaces, it could not tackle problems defined over continuous state spaces. By contrast, in this paper we show how to use approximate MDP solution methods for taking decisions in this domain without the need of discretizing the state space. Specifically, we employ fitted value iteration, a sampling-based approximation method that is known to be well behaved. By so doing, we generalize our factored MDP solution method to continuous state spaces. We evaluate our approach using a variety of basis functions over different state sample sizes, and compare its performance to that of our original “exact” value iteration algorithm. Our generic approximation method is shown to exhibit stable performance in terms of accumulated reward, which for certain basis functions reaches 98% of that gathered by the exact algorithm.