Deep inverse reinforcement learning by logistic regression

Abstract

This study proposes model-free deep inverse reinforcement learning to find nonlinear reward function structures. It is based on our previous method that exploits the fact that the log of the ratio between an optimal state transition and a baseline one is given by a part of reward and the difference of the value functions under linearly solvable Markov decision processes and reward and value functions are estimated by logistic regression. However, reward is assumed to be a linear function whose basis functions are prepared in advance. To overcome this limitation, we employ deep neural network frameworks to implement logistic regression. Simulation results show our method is comparable to model-based previous methods with less computing effort in the Objectworld benchmark. In addition, we show the optimal policy, which is trained with the shaping reward using the estimated reward and value functions, outperforms the policies that are used to collect data in the game of Reversi.