A Multi-FPGA Scalable Framework for Deep Reinforcement Learning Through Neuroevolution

Abstract

The application of Deep Neural Networks (DNN) for reinforcement learning has proven effective in solving complex problems, such as playing video games or training robots to perform human tasks. Training based on reinforcement implies the continuous interaction of the agent powered by the DNN and the environment, vanishing the typical separation between the training and inference stages in deep learning. However, the high memory and accuracy requirements of gradient-based training algorithms prevent using FPGAs for these applications. As an alternative, this work demonstrates the feasibility of using Evolutionary Algorithms (EA) for training DNNs and their usage in reinforcement learning scenarios. Unlike backpropagation, EA-based training of neural networks, referred to as neuroevolution, can be effectively implemented on FPGAs. Moreover, this paper shows how the inherent parallelism of EAs can be effectively exploited in multi-FPGA scenarios to accelerate the learning process. The proposed FPGA-based neuroevolutionary framework has been validated by building a system capable of learning autonomously to play the Pong Atari game in less than 25 generations.