Falsification has emerged as an important tool for simulation-based verification of autonomous systems. In this paper, we present extensions to the Scenic scenario specification language and VerifAI toolkit that improve the scalability of sampling-based falsification methods by using parallelism and extend falsification to multi-objective specifications. We first present a parallelized framework that is interfaced with both the simulation and sampling capabilities of Scenic and the falsification capabilities of VerifAI, reducing the execution time bottleneck inherently present in simulation-based testing. We then present an extension of VerifAI's falsification algorithms to support multi-objective optimization during sampling, using the concept of rulebooks to specify a preference ordering over multiple metrics that can be used to guide the counterexample search process. Lastly, we evaluate the benefits of these extensions with a comprehensive set of benchmarks written in the Scenic language.
翻译:伪造工作已成为对自主系统进行模拟核查的一个重要工具。在本文中,我们介绍了对 " 现场情景规格 " 语言和 " VerifAI " 工具包的扩展,这些扩展通过使用平行方法,将基于取样的伪造方法的可缩放性扩大到多目标规格;我们首先提出了一个平行框架,与 " 现场 " 和 " VerifAI " 的模拟和取样能力以及伪造能力相结合,减少了在模拟测试中固有的执行时间瓶颈;然后,我们介绍了 " VerifAI " 的伪造算法的扩展,以支持取样过程中的多目标优化,利用规则手册的概念,具体规定对可用于指导反抽样搜索过程的多指标的优先排序;最后,我们用一套以 " 现场语言 " 写成的全面基准评估这些延期的好处。