Evaluating safety performance in a resource-efficient way is crucial for the development of autonomous systems. Simulation of parameterized scenarios is a popular testing strategy but parameter sweeps can be prohibitively expensive. To address this, we propose HiddenGems: a sample-efficient method for discovering the boundary between compliant and non-compliant behavior via active learning. Given a parameterized scenario, one or more compliance metrics, and a simulation oracle, HiddenGems maps the compliant and non-compliant domains of the scenario. The methodology enables critical test case identification, comparative analysis of different versions of the system under test, as well as verification of design objectives. We evaluate HiddenGems on a scenario with a jaywalker crossing in front of an autonomous vehicle and obtain compliance boundary estimates for collision, lane keep, and acceleration metrics individually and in combination, with 6 times fewer simulations than a parameter sweep. We also show how HiddenGems can be used to detect and rectify a failure mode for an unprotected turn with 86% fewer simulations.
翻译:以资源效率高的方式评估安全性能对于开发自主系统至关重要。 模拟参数化假设情景是一个广受欢迎的测试策略,但参数扫描费用可高得令人望而却步。 为了解决这个问题,我们提议隐藏Gems:一种通过积极学习发现合规行为与不合规行为之间的界限的样本高效方法。鉴于一种参数化假设情景、一种或多种合规度指标以及一个模拟符,隐藏Gems绘制了该假设情景中符合和不合规的领域。该方法使得能够进行关键测试案例识别,对测试中系统的不同版本进行比较分析,以及核实设计目标。我们用自动车辆前的公用行横越线来评估隐藏Gems的情景,并获得单个和组合的合规边界估计,其模拟比参数扫描少6倍。我们还展示了隐藏Gems如何用86%的模拟来探测和纠正无保护的转弯的故障模式。