A Reinforcement Learning Approach for Transient Control of Liquid Rocket Engines

Abstract

Nowadays, liquid rocket engines use closed-loop control at most near-steady operating conditions. The control of the transient phases is traditionally performed in open loop due to highly nonlinear system dynamics. This situation is unsatisfactory, in particular for reusable engines. The open-loop control system cannot provide optimal engine performance due to external disturbances or the degeneration of engine components over time. In this article, we study a deep reinforcement learning approach for optimal control of a generic gas-generator engine's continuous startup phase. It is shown that the learned policy can reach different steady-state operating points and convincingly adapt to changing system parameters. Compared to carefully tuned open-loop sequences and proportional-integral-derivative (PID) controllers, the deep reinforcement learning controller achieves the highest performance. In addition, it requires only minimal computational effort to calculate the control action, which is a big advantage over approaches that require online optimization, such as model predictive control.