Developing and Testing a New Reinforcement Learning Toolkit with Unreal Engine

Abstract

In this work we tried to overcome the main limitations that can be found in current state-of-the-art development and benchmarking RL platforms, namely the lack of a user interface, a closed-box approach to scenarios, the lack of realistic environments and the difficulty in extending the obtained results in real applications, by introducing a new development framework for reinforcement learning built over the graphics engine Unreal Engine 4. Unreal Reinforcement Learning Toolkit (URLT) was developed with the idea of being modular, flexible, and easy to use even for non-expert users. To do this, we have developed flexible and modular APIs, through which it’s possible to setup the major learning techniques. Using these APIs, users can define all the elements of a RL problem, such as agents, algorithms, tasks, and scenarios, and to combine them with each other to always have new solutions. By taking advantage of the editor’s UI, users can select and execute existing scenarios and change the parameters of agents and tasks without to recompile the code. Users also have the possibility to create new scenarios from scratch using an intuitive level editor. Furthermore, task design is made accessible to non-expert users using a node-oriented visual programming system called Blueprint. To validate the tool, we produced a starter pack containing a suite of state-of-the-art RL algorithms, some example scenarios, a small library of props and a couple of trainable agents. Moreover, we ran an evaluation test with users in which the latter were required to try URLT and competing software (OpenAI Gym), then to evaluate both using a questionnaire. The obtained results showed a general preference for URLT in all key parameters of the test.