Learning continuous muscle control for a multi-joint arm by extending proximal policy optimization with a liquid state machine

Abstract

There have been many advances in the field of reinforcement learning in continuous control problems. Usually, these approaches use deep learning with artificial neural networks for approximation of policies and value functions. In addition, there have been interesting advances in spiking neural networks, towards a more biologically plausible model of the neurons and the learning mechanisms. We present an approach to learn continuous muscle control of a multi joint arm. We use reinforcement learning for a target reaching task, which can be modeled as partially observable markov decision processes. We extend proximal policy optimization with a liquid state machine (LSM) for state representation to achieve better performance in the target reaching task. The results show that we are able to learn to control the arm after training the readout of the LSM with reinforcement learning. The input current encoding used for encoding the state is enough to have a good projection into a higher dimensional space of the LSM. The results also show that we are able to learn a linear readout, which is equivalent to a one-layer neural network to learn to control the arm. We show that there are clear benefits of training the readouts of a LSM with reinforcement learning. These results can lead to demonstrate the benefits of using a LSM as a drop-in state transformation in general.