Generating Critical Test Scenarios for Autonomous Driving Systems via Influential Behavior Patterns

Abstract

Autonomous Driving Systems (ADSs) are safety-critical, and must be fully tested before being deployed on real-world roads. To comprehensively evaluate the performance of ADSs, it is essential to generate various safety-critical scenarios. Most of existing studies assess ADSs either by searching high-dimensional input space, or using simple and pre-defined test scenarios, which are not efficient or not adequate. To better test ADSs, this paper proposes to automatically generate safety-critical test scenarios for ADSs by influential behavior patterns, which are mined from real traffic trajectories. Based on influential behavior patterns, a novel scenario generation technique, CRISCO, is presented to generate safety-critical scenarios for ADSs testing. CRISCO assigns participants to perform influential behaviors to challenge the ADS. It generates different test scenarios by solving trajectory constraints, and improves the challenge of those non-critical scenarios by adding participants' behavior from influential behavior patterns incrementally. We demonstrate CRISCO on an industrial-grade ADS platform, Baidu Apollo. The experiment results show that our approach can effectively and efficiently generate critical scenarios to crash ADS, and it exposes 13 distinct types of safety violations in 12 hours. It also outperforms two state-of-art ADS testing techniques by exposing more 5 distinct types of safety violations on the same roads.