Out-of-distribution (OOD) detection aims to identify test examples that do not belong to the training distribution and are thus unlikely to be predicted reliably. Despite a plethora of existing works, most of them focused only on the scenario where OOD examples come from semantic shift (e.g., unseen categories), ignoring other possible causes (e.g., covariate shift). In this paper, we present a novel, unifying framework to study OOD detection in a broader scope. Instead of detecting OOD examples from a particular cause, we propose to detect examples that a deployed machine learning model (e.g., an image classifier) is unable to predict correctly. That is, whether a test example should be detected and rejected or not is ``model-specific''. We show that this framework unifies the detection of OOD examples caused by semantic shift and covariate shift, and closely addresses the concern of applying a machine learning model to uncontrolled environments. We provide an extensive analysis that involves a variety of models (e.g., different architectures and training strategies), sources of OOD examples, and OOD detection approaches, and reveal several insights into improving and understanding OOD detection in uncontrolled environments.
翻译:离群检测旨在确定训练分布之外的测试样例,因此很可能无法可靠地预测。尽管已经有大量现有的研究工作,但大多数工作只关注了离群样例来自语义转换(如看不见的类别)的情况,而忽略了其他可能的原因(如协变量转换)。在本文中,我们提出了一个新的、统一的框架来研究更广泛的离群检测。我们建议不是从特定的原因来检测离群样例,而是从 deployed 机器学习模型(例如图像分类器)无法正确预测的样例开始。也就是说,是否应该检测并拒绝测试样例是“模型特定”的。我们展示了这个框架统一了由语义转移和协变量转换引起的离群样例的检测,并密切关注将机器学习模型应用到不受控制的环境的问题。我们进行了广泛的分析,涉及各种模型(例如不同的体系结构和训练策略)、离群样例的来源和离群检测方法,并揭示了一些改进和理解离群检测在不受控制的环境中的见解。