Attention-based deep Q-network in complex systems

Abstract

In recent years, Deep Reinforcement Learning (DRL) has achieved great successes in many large scale applications, e.g., the Deep Q-Network (DQN) surpasses the level of professional human players in most of the challenging Atari 2600 games. As DQN transforms the whole input frames into some feature vectors by using convolutional neural networks (CNNs) at each decision step, all objects in the system are treated equally in the process of the feature extraction. However, in reality, for complex systems where many objects exist, the optimal action taken by the agent may only be affected by some important objects, which may lead to inefficiency or poor performance of DQN. In order to alleviate this problem, in this paper, we introduce two approaches that integrate global and local attention mechanisms respectively into the DQN model. For the approach with global attention, the agent is able to focus on all objects to varying degrees; for the approach with local attention, the agent is allowed to focus only on a few objects of great importance with the result that a better strategy can be learned by the agent. The performance of our proposed approaches are demonstrated on some benchmark domains. Source code is available at https://github.com/DMU-XMU/Attention-based-DQN.