Deep learning methods have boosted the adoption of NLP systems in real-life applications. However, they turn out to be vulnerable to distribution shifts over time which may cause severe dysfunctions in production systems, urging practitioners to develop tools to detect out-of-distribution (OOD) samples through the lens of the neural network. In this paper, we introduce TRUSTED, a new OOD detector for classifiers based on Transformer architectures that meets operational requirements: it is unsupervised and fast to compute. The efficiency of TRUSTED relies on the fruitful idea that all hidden layers carry relevant information to detect OOD examples. Based on this, for a given input, TRUSTED consists in (i) aggregating this information and (ii) computing a similarity score by exploiting the training distribution, leveraging the powerful concept of data depth. Our extensive numerical experiments involve 51k model configurations, including various checkpoints, seeds, and datasets, and demonstrate that TRUSTED achieves state-of-the-art performances. In particular, it improves previous AUROC over 3 points.
翻译:深层的学习方法推动了在现实应用中采用NLP系统,然而,这些方法最终容易在一段时间内造成生产系统严重机能失调的分布变化,敦促从业者开发工具,通过神经网络的透镜,探测分布(OOOD)样本。在本文中,我们引入了基于符合操作要求的变异结构的分类器新的OOOD探测器TRUTED:它不受监督和快速计算。TRUSTED的效率取决于所有隐蔽层都携带相关信息以探测OOOD实例的富有成果的想法。基于这一想法,TRUSTED(T)包含:(一) 汇集这一信息,(二) 通过利用培训分布,利用强大的数据深度概念计算相似的分数。我们广泛的数字实验涉及51k模型配置,包括各种检查站、种子和数据集,并表明TRUSTED取得了最新表现。特别是它改进了以前的AUROC的三点。