Model-Based Data-Efficient Reinforcement Learning for Active Pantograph Control in High-Speed Railways

Abstract

The active pantograph is a promising technology to suppress contact force (CF) fluctuation in pantograph-catenary systems (PCSs). Recently, the rapid development of reinforcement learning (RL) techniques has dramatically facilitated complex system controllers' design. However, the low data efficiency problem is fatal because data collection is costly. In this article, we propose the ensemble Q-functions model-based RL (EQ-MBRL) algorithm to achieve data-efficient RL. First, we introduce an ensemble probabilistic neural network (EPNN) to estimate the distribution and uncertainty of the dynamics model and adopt multistep loss to constrain the accumulation error in the long-length model rollout. Second, we employ a short-term rollout of the model to trade off the ease of data generation and the error of the model-generated data. Finally, we propose ensemble Q-functions and in-target minimization techniques to help stabilize the training process of value functions and improve the accuracy of value estimation. In addition, we discussed the appropriate model-based rollout length and explored the performance of network update rates with different strategies. The experimental results demonstrate that the proposed approach outperforms compared algorithms and delivers a state-of-the-art performance on the PCS benchmark. The controller learned robust motion patterns using only 50k collected transitions, which was more than ten times faster than compared baseline.