The usage of deep neural networks in safety-critical systems is limited by our ability to guarantee their correct behavior. Runtime monitors are components aiming to identify unsafe predictions and discard them before they can lead to catastrophic consequences. Several recent works on runtime monitoring have focused on out-of-distribution (OOD) detection, i.e., identifying inputs that are different from the training data. In this work, we argue that OOD detection is not a well-suited framework to design efficient runtime monitors and that it is more relevant to evaluate monitors based on their ability to discard incorrect predictions. We call this setting out-ofmodel-scope detection and discuss the conceptual differences with OOD. We also conduct extensive experiments on popular datasets from the literature to show that studying monitors in the OOD setting can be misleading: 1. very good OOD results can give a false impression of safety, 2. comparison under the OOD setting does not allow identifying the best monitor to detect errors. Finally, we also show that removing erroneous training data samples helps to train better monitors.
翻译:运行时间监测器是旨在查明不安全预测的部件,并在可能导致灾难性后果之前抛弃这些预测。最近一些运行时间监测工作的重点是在分配之外探测(OOD),即查明与培训数据不同的投入。在这项工作中,我们认为OOOD检测不是一个设计高效运行时间监测器的合适框架,根据监测器放弃错误预测的能力来评估监测器更为相关。我们称之为模型范围探测,并与OOOD讨论概念差异。我们还对文献中的流行数据集进行了广泛的实验,以表明在OOOD环境中研究监测器可能会产生误导:1. 非常好的OOD结果可能造成安全错误印象,2. 在OOD环境下进行比较无法确定最佳监测器来检测错误。最后,我们还表明,删除错误的培训数据样本有助于培训更好的监测器。