Testing black-box perceptual-control systems in simulation faces two difficulties. Firstly, perceptual inputs in simulation lack the fidelity of real-world sensor inputs. Secondly, for a reasonably accurate perception system, encountering a rare failure trajectory may require running infeasibly many simulations. This paper combines perception error models -- surrogates for a sensor-based detection system -- with state-dependent adaptive importance sampling. This allows us to efficiently assess the rare failure probabilities for real-world perceptual control systems within simulation. Our experiments with an autonomous braking system equipped with an RGB obstacle-detector show that our method can calculate accurate failure probabilities with an inexpensive number of simulations. Further, we show how choice of safety metric can influence the process of learning proposal distributions capable of reliably sampling high-probability failures.
翻译:在模拟中测试黑箱感知控制系统面临两个困难。 首先,模拟中的感知输入缺乏真实世界感官输入的真实性。 其次,对于合理准确的感知系统来说,遇到罕见的失败轨迹可能需要运行许多不可行的模拟。 本文结合了感知错误模型 -- -- 以传感器为基础的检测系统代孕 -- -- 以及国家依赖的适应重要性抽样。 这使我们能够有效地评估模拟中真实世界感知控制系统罕见的失灵概率。 我们用安装了RGB障碍检测器的自动制动系统进行的实验表明,我们的方法可以用廉价的模拟数来计算准确失败概率。 此外,我们展示了安全度指标的选择会如何影响能够可靠地取样高概率故障的学习建议分发过程。