Q-Tree: Automatic construction of hierarchical state representation for reinforcement learning

Abstract

A primary challenge of agent-based reinforcement learning in complex and uncertain environments is escalating computational complexity with the number of the states. Hierarchical, or tree-based, state representation provides a promising approach to complexity reduction through clustering and sequencing of similar states. We introduce the Q-Tree algorithm to utilize the data history of state transition information to automatically construct such a representation and to obtain a series of linear separations between state clusters to facilitate learning. Empirical results for the canonical PuddleWorld problem are provided to validate the proposed algorithm; extensions of the PuddleWorld problem obtained by adding random noise dimensions are solved by the Q-Tree algorithm, while traditional tabular Q-learning cannot accommodate random state elements within the same number of learning trials. The results show that the Q-Tree algorithm can reject state dimensions that do not aid learning by analyzing weights of all linear classifiers for a hierarchical state representation. © Springer-Verlag Berlin Heidelberg 2012.