Constructing Action Space from Basis Functions for Motion Acquisition of Robots by Reinforcement Learning

Abstract

Discrete action sets are often used in many reinforcement learning (RL) applications in robot control, since such sets are compatible with many RL methods and sophisticated architectures, such as Q(λ)-learning [1] and the Dyna. However, one of the problems is the absence of general principles on designing a discrete action set for robot control in higher dimensional input space. In this paper, we propose a discrete action set DCOB which is generated from the given basis functions (BFs) for approximating a value function. Though the DCOB is a discrete set, it has an ability to acquire high performance. Moreover, we utilize a method that generates a set of BFs based on the dynamics of the robot to reduce the number of the BFs. This way also makes the DCOB compact. Thus, the DCOB is compact and has an ability to acquire performance. Moreover, we also propose a method WF-DCOB, where the wire-fitting [2] is utilized to learn within a continuous action space which the DCOB discretizes. The purpose of the WF-DCOB is to evaluate the possibility of acquiring higher performance. Our proposition in the WF-DCOB is to constrain and initialize the parameters to relax the instability of the wire-fitting. We apply the proposed methods for a humanoid robot to learn crawling motion. The simulation results demonstrate outstanding advantages of the proposed method both in learning speed and ability to acquire performance, compared to conventional action space.