TARGET: Traffic Rule-based Test Generation for Autonomous Driving Systems

Recent accidents involving self-driving cars call for extensive testing efforts to improve the safety and robustness of autonomous driving. However, constructing test scenarios for autonomous driving is tedious and time-consuming. In this work, we develop an end-to-end test generation framework called TARGET, which automatically constructs test scenarios from human-written traffic rules in an autonomous driving simulator. To handle the ambiguity and sophistication of natural language, TARGET uses GPT-3 to extract key information related to the test scenario from a traffic rule and represents the extracted information in a test scenario schema. Then, TARGET synthesizes the corresponding scenario scripts to construct the test scenario based on the scenario representation. We have evaluated TARGET on four autonomous driving systems, 18 traffic rules, and 8 road maps. TARGET can successfully generate 75 test scenarios and detect 247 traffic rule violations. Based on the violation logs (e.g., waypoints of ego vehicles), we are able to identify three underlying issues in these autonomous driving systems, which are either confirmed by the developers or the existing bug reports.