Safety Analysis of Autonomous Driving Systems: A Simulation-Based Runtime Verification Approach

Abstract

Ensuring the safety of autonomous driving systems (ADSs) through rigorous verification in simulated environments is crucial before real-world deployment. However, using simulation environments for ADS testing and verification poses several challenges, including specifying various behaviors of traffic participants and collecting comprehensive real-time data to verify the ADS. To address these challenges, we propose a framework for runtime verification of ADSs, focusing on Autoware, a leading ADS. This framework integrates AWSIM-Script, a scripting language for defining traffic scenarios; Runtime Monitor, a tool to record real-time data during the simulation; and AW-Checker, a Linear Temporal Logic-based property checker for verifying safety requirements. Unlike prior research that primarily focuses on generating critical scenarios, we leverage a well-established ADS safety standard from the Japan Automobile Manufacturers Association and adopt its systematic methodology for safety assessment. We conducted a series of experiments focused on nonintersection road geometry to evaluate Autoware's capability in handling different traffic disturbances such as cut-in, cut-out, and deceleration scenarios. The results revealed that, compared to the competent and careful driver model, which represents the minimum safety requirements for ADSs, Autoware failed to prevent collisions in some cases, particularly during high-speed scenarios and fast lateral movements by other vehicles.