Model Predictive Control With Reinforcement Learning-Based Speed Profile Generation in Racing Simulator

Abstract

Model Predictive Control (MPC) is a widely used optimal control strategy, particularly effective in managing complex constraints. It excels at optimizing performance within feasible limits, such as minimizing lap times in vehicle racing. However, its effectiveness can be hindered by computational burdens and model inaccuracies. To address these challenges, MPC problems are often simplified, and parameters are manually tuned through iterative adjustments - a process that is both time-consuming and labor-intensive. This paper introduces a Reinforcement Learning-based Speed Profile Model Predictive Control (RLSP-MPC) approach, which leverages deep reinforcement learning (DRL) to approximate global optima via an exploration-exploitation framework, eliminating the need for manual tuning. The RLSP-MPC method reformulates the traditional MPC speed optimization task into a DRL-enhanced speed profile tracking problem, reducing parameter dependency and improving lap times. The DRL network provides boundary constraints and a reference velocity profile to MPC, which then reconstructs the trajectory while adhering to these constraints. Polynomial functions are employed to reduce the dimensionality of the network output in velocity profile design. Extensive experiments conducted using the F1Tenth Simulator confirm that the proposed method effectively integrates the strengths of MPC and DRL, achieving superior lap times and enhanced trajectory tracking performance while maintaining real-time computational feasibility. Specifically, the proposed method reduces lap times by an average of approximately 22.07% compared to conventional MPC and 14.52% compared to LMPC across multiple racing tracks, demonstrating consistent and statistically significant performance improvements.