Hierarchical Reinforcement Learning for Autonomous Decision Making and Motion Planning of Intelligent Vehicles

Abstract

Autonomous decision making and motion planning in complex dynamic traffic environments, such as left-turn without traffic signals and multi-lane merging from side-ways, are still challenging tasks for intelligent vehicles. It is difficult to generate optimized behavior decisions while considering the motion capabilities and dynamic properties of intelligent vehicles. Aiming at the above problems, this article proposes a hierarchical reinforcement learning approach for autonomous decision making and motion planning in complex dynamic traffic scenarios. The proposed approach consists of two layers. At the higher layer, a kernel-based least-squares policy iteration algorithm with uneven sampling and pooling strategy (USP-KLSPI) is presented for solving the decision-making problems. The motion capabilities of the ego vehicle and the surrounding vehicles are evaluated with a high-fidelity dynamic model in the decision-making layer. By doing so, the consistency between the decisions generated at the higher layer and the operations in the lower planning layer can be well guaranteed. The lower layer addresses the motion-planning problem in the lateral direction using a dual heuristic programming (DHP) algorithm learned in a batch-mode manner, while the velocity profile in the longitudinal direction is inherited from the higher layer. Extensive simulations are conducted in complex traffic conditions including left-turn without traffic signals and multi-lane merging from side-ways scenarios. The results demonstrate the effectiveness and efficiency of the proposed approach in realizing optimized decision making and motion planning in complex environments.