Trade-Offs in Navigation Problems Using Value-Based Methods

Abstract

Deep Q-Networks (DQNs) have shown remarkable results over the last decade in scenarios ranging from simple 2D fully observable short episodes to partially observable, graphically intensive, and complex tasks. However, the base architecture of a vanilla DQN presents several shortcomings, some of which were mitigated by new variants focusing on increased stability, faster convergence, and time dependencies. These additions, on the other hand, bring increased costs in terms of the required memory and lengthier training times. In this paper, we analyze the performance of state-of-the-art DQN families in a simple partially observable mission created in Minecraft and try to determine the optimal architecture for such problem classes in terms of the cost and accuracy. To the best of our knowledge, the analyzed methods have not been tested on the same scenario before, and hence a more in-depth comparison is required to understand the real performance improvement they provide better. This manuscript also offers a detailed overview of state-of-the-art DQN methods, together with the training heuristics and performance metrics registered during the proposed mission, allowing researchers to select better-suited models to solving future problems. Our experiments show that Double DQN networks are capable of handling partially observable scenarios gracefully while maintaining a low hardware footprint, Recurrent Double DQNs can be a good candidate even when the resources must be restricted, and double-dueling DQNs are a well-performing middle ground in terms of their cost and performance.