Hierarchical reinforcement learning-based traffic signal control

Abstract

Efficient traffic light control is a critical issue in urban transportation systems. Recently, deep reinforcement learning (DRL) has gained popularity as a method for real-time traffic light control. However, state-of-the-art DRL-based systems typically construct individual RL agents for each intersection, each focusing on local objectives, potentially undermining overall global traffic efficiency. To overcome this limitation, we propose SHLight (Sample selection-based Hierarchical traffic Light control method), an innovative RL method featuring a hierarchical framework comprising a manager and multiple workers. Specifically, we partition the traffic network into multiple regions, assigning a manager agent to oversee each region, with traffic signal controllers acting as workers. The manager sets goals for the workers based on the overarching global objective, empowering the workers to achieve these goals while concurrently optimizing local traffic efficiency. Our method introduces several key innovations: it employs importance sampling to mitigate the non-stationarity problem inherent in hierarchical reinforcement learning, and it incorporates auxiliary actions to enhance the observability of the DRL model. Our extensive simulations conducted on both synthetic and real-world road networks demonstrate that SHLight outperforms the state-of-the-art models, showcasing reduced queue lengths, waiting times, and delays.