ST-D3QN: Advancing UAV Path Planning with an Enhanced Deep Reinforcement Learning Framework in Ultra-Low Altitudes

Abstract

Unmanned aerial vehicles (UAVs) have increasingly become indispensable instruments for surveillance, reconnaissance, and various missions due to their advanced capabilities. In intricate and dynamic operational contexts, especially during ultra-low altitude missions, inadequate communication and coordination impede UAVs' capacity to avoid other UAVs and moving objects. In this instance, attaining accurate path planning for UAVs constitutes a significant technological problem that requires immediate attention. This paper introduces an innovative algorithm, ST-D3QN, which leverages deep reinforcement learning (DRL) to optimize UAV trajectories based on optimizing the Dueling Double Deep Q-Network (D3QN) and A∗ (A-star) algorithm. Our approach refines the optimization of sparse rewards by toggling between the two methods, thereby maximizing the long-term benefits for UAVs. The greedy strategy has been refined to prevent the algorithm from becoming ensnared in local optima. Our proposed method is benchmarked against state-of-the-art path planning algorithms through meticulously designed simulation tests. The experimental results indicate that the ST-D3QN algorithm achieves the fastest convergence rate during training and excels in optimizing sparse rewards. It also significantly reduces the risk of collisions and enhances the success rates of UAV path planning, establishing itself as an efficient and effective solution for determining the shortest path for UAVs in complex environments.