Skill Learning for Long-Horizon Sequential Tasks

Abstract

Solving long-horizon problems is a desirable property in autonomous agents. Learning reusable behaviours can equip the agent with this property, allowing it to adapt them when performing various real-world tasks. Our approach for learning these behaviours is composed of three modules, operating in two separate timescales and it uses a hierarchical model with both discrete and continuous variables. This modular structure allows an independent training process for each stage. These stages are organized using a two-level temporal hierarchy. The first level contains the planner, responsible for issuing the skills that should be executed, while the second level executes the skill. In this latter level, to achieve the desired skill behaviour, the discrete skill is converted to a continuous vector that contains information regarding which environment change must occur. With this approach, we aimed to solve long-horizon sequential tasks with delayed rewards. Contrary to existing work, our method uses both variable types to allow an agent to learn high-level behaviours consisting of an interpretable set of skills. This method allows to compose the discrete skills easily, while keeping the flexibility, provided by the continuous representations, to execute them in several different ways. Using a 2D scenario where the agent has to catch a set of objects in a specific order, we demonstrate that our approach is scalable to scenarios with increasingly longer tasks.