A sampling hyperbelief optimization technique for stochastic systems

Abstract

Uncertainty plays a dramatic role not only on the quality of the optimal solution of POMDP system, but also on the computational complexity of finding the optimal solution, with a worst case running time that is exponential in the length of the time horizon for the exact solution. However, given the importance of finding optimal or nearly optimal solutions for systems subject to uncertainty, numerous researchers have developed approaches to approximate POMDP systems to overcome this limitation (refer to [22, Ch. 15 & 16] for a survey of such approaches). The majority of these methods are for discounted, infinite-horizon problems. Moreover, many of these techniques must reperform all the computational effort when the objective function changes. A central theme of almost all approximation techniques is to reduce the set of possibilities to be evaluated, whether simplifying the representation of the belief or by the simplifying the value function. Drawing on insights offered in [8] about why belief sampling techniques (such as [21, 15, 17, 18, 19, 20]) are so effective, we develop an alternative method that is inspired by graph sampling-based methods (e.g., [10]). In [6], we introduce the notion of hyperfiltering, which evolves forward into future stages the probability function over the belief, or the hyperbelief.We refer to the space of probability functions over the belief as the hyperbelief space. Interestingly, the evolution of a system over the hyperbelief space is deterministic. Thus, we can find the optimal plan in the hyperbelief space using an approach derived from standard search techniques. © 2009 Springer-Verlag.