Hierarchical Reinforcement Learning with Clustering Abstract Machines

Abstract

Hierarchical reinforcement learning (HRL) is another step towards the convergence of learning and planning methods. The resulting reusable abstract plans facilitate both the applicability of transfer learning and increasing of resilience in difficult environments with delayed rewards. However, on the way of the practical application of HRL, especially in robotics, there are a number of difficulties, among which the key is a semi-manual task of the creation of the hierarchy of actions, which the agent uses as a pre-trained scheme. In this paper, we present a new approach for simultaneous constructing and applying the hierarchy of actions and sub-goals. In contrast to prior efforts in this direction, the method is based on a united loop of clustering of the environment’s states observed by the agent and allocation of sub-targets by the modified bottleneck method for constructing of abstract machines hierarchy. The general machine is built using the so-called programmable schemes, which are quite universal for the organization of transfer learning for a wide class of tasks. A particular abstract machine is assigned for each set of clustered states. The goal of each machine is to reach one of the found bottleneck states and then get into another cluster. We evaluate our approach using a standard suite of experiments on a challenging planning problem domain and show that our approach facilitates learning without prior knowledge.