Traffic Signal Control under Mixed Traffic with Connected and Automated Vehicles: A Transfer-Based Deep Reinforcement Learning Approach

Abstract

Backgrounds: The traffic signal control (TSC) system could be more intelligently controlled by deep reinforcement learning (DRL) and information provided by connected and automated vehicles (CAVs). However, the direct training procedure of the DRL is time-consuming and hard to converge. Methods: This study improves the training efficiency of the deep Q network (DQN) by transferring the well-trained action policy of a previous DQN model into a target model under similar traffic scenarios. Different reward parameters, exploration rates, and action step lengths are tested. The performance of the transfer-based DQN-TSC is analyzed by considering different traffic demands and market penetration rates (MPRs) of CAVs. The information level requirements of the DQN-TSC are also investigated. Results: Compared to directly trained DQN, transfer-based models could improve both the training efficiency and model performance. In high traffic scenarios with a 100% MPR of CAVs, the total waiting time, CO2 emission, and fuel consumption in the transfer-based TSC decrease about 38%, 34%, and 34% compared to pre-timed signal schemes. Also, the transfer-based TSC system requires more than 20% to 40% MPRs of CAVs under different traffic demands to perform better than pre-timed signal schemes. Conclusions: The proposed model could improve both the traffic performance of the TSC system and the training efficiency of the DQN model. The insights of this study should be helpful to planners and engineers in designing intelligent signal intersections and providing guidance for engineering applications of the DQN TSC systems.