Road network traffic control reinforcement learning algorithms based on Nash-Stackelberg hierarchical game model

Abstract

To solve the problem of computational complexity for the Nash equilibrium caused by multi-intersection game, with the consideration of the importance of different intersections in the road network and the game relationships between the intersections, a Nash-Stackelberg hierarchical game(NSHG)model is proposed which takes into account the traffic control strategies between and within the subareas of the road network, with the important intersections in the two sub-areas as the game subject at the upper layer and the secondary intersections as the game subject at the lower layer. Two multi-agent reinforcement learning(MARL)algorithms, NSHG-based Q learning(NSHG-QL)algorithm and NSHG-based deep Q network(NSHG-DQN)algorithm are proposed. In the experiments, the signals are controlled using NSHG-QL and NSHG-DQN algorithms in the road network environment built by SUMO simulation software and compared with the base game model solution algorithm. The experimental results show that, NSHG-QL and NSHG-DQN algorithms can reduce the average travel time and the time loss of vehicles at the intersections, and increase the average speed. NSHG model can coordinate the secondary intersections to make optimal strategy selections on the basis of satisfying the upper-layer game between the important intersections. Moreover, the MARL algorithms based on the hierarchical game model can significantly improve learning performance and convergence.