Pretrained Transformers achieve remarkable performance when training and test data are from the same distribution. However, in real-world scenarios, the model often faces out-of-distribution (OOD) instances that can cause severe semantic shift problems at inference time. Therefore, in practice, a reliable model should identify such instances, and then either reject them during inference or pass them over to models that handle another distribution. In this paper, we develop an unsupervised OOD detection method, in which only the in-distribution (ID) data are used in training. We propose to fine-tune the Transformers with a contrastive loss, which improves the compactness of representations, such that OOD instances can be better differentiated from ID ones. These OOD instances can then be accurately detected using the Mahalanobis distance in the model's penultimate layer. We experiment with comprehensive settings and achieve near-perfect OOD detection performance, outperforming baselines drastically. We further investigate the rationales behind the improvement, finding that more compact representations through margin-based contrastive learning bring the improvement. We release our code to the community for future research.
翻译:培训前的变异器在培训和测试数据来自同一分布时取得显著的性能。 但是,在现实世界中,模型往往会面临分配外(OOD)情况,在推论时间可能造成严重的语义转变问题。 因此,在实践中,可靠的模型应该识别这类情况,然后在推断期间拒绝它们,或者将其传递给处理另一分布的模型。在本文中,我们开发了一种不受监督的OOOD检测方法,在培训中只使用分布(ID)数据。我们建议微调变异的变异器损失,这可以改善表述的紧凑性,使OOD情况与身份特征发生更好的区别。然后可以精确地检测出这些OOOD情况,在模型的侧侧面层使用Mahalanobis距离进行测试,并实现接近于 OOODD的检测性能,比基线要快得多。我们进一步调查改进背后的理由,通过边际对比学习发现更紧凑的表达方式带来改进。我们向社区公布我们的代码,以便将来进行研究。