Adaptive Temporal Reinforcement Learning for Mapping Complex Maritime Environmental State Spaces in Autonomous Ship Navigation

Abstract

The autonomous decision-making model for ship navigation requires extensive interaction and trial-and-error in real, complex environments to ensure optimal decision-making performance and efficiency across various scenarios. However, existing approaches still encounter significant challenges in addressing the temporal features of state space and tackling complex dynamic collision avoidance tasks, primarily due to factors such as environmental uncertainty, the high dimensionality of the state space, and limited decision robustness. This paper proposes an adaptive temporal decision-making model based on reinforcement learning, which utilizes Long Short-Term Memory (LSTM) networks to capture temporal features of the state space. The model integrates an enhanced Proximal Policy Optimization (PPO) algorithm for efficient policy iteration optimization. Additionally, a simulation training environment is constructed, incorporating multi-factor coupled physical properties and ship dynamics equations. The environment maps variables such as wind speed, current velocity, and wave height, along with dynamic ship parameters, while considering the International Regulations for Preventing Collisions at Sea (COLREGs) in training the autonomous navigation decision-making model. Experimental results demonstrate that, compared to other neural network-based reinforcement learning methods, the proposed model excels in environmental adaptability, collision avoidance success rate, navigation stability, and trajectory optimization. The model’s decision resilience and state-space mapping align with real-world navigation scenarios, significantly improving the autonomous decision-making capability of ships in dynamic sea conditions and providing critical support for the advancement of intelligent shipping.