Distribution shifts are problems where the distribution of data changes between training and testing, which can significantly degrade the performance of a model deployed in the real world. Recent studies suggest that one reason for the degradation is a type of overfitting, and that proper regularization can mitigate the degradation, especially when using highly representative models such as neural networks. In this paper, we propose a new regularization using the supervised contrastive learning to prevent such overfitting and to train models that do not degrade their performance under the distribution shifts. We extend the cosine similarity in contrastive loss to a more general similarity measure and propose to use different parameters in the measure when comparing a sample to a positive or negative example, which is analytically shown to act as a kind of margin in contrastive loss. Experiments on benchmark datasets that emulate distribution shifts, including subpopulation shift and domain generalization, demonstrate the advantage of the proposed method over existing regularization methods.
翻译:基于异质相似度的有监督对比学习用于分布漂移
分布漂移是训练集和测试集数据分布发生变化的问题,这可能会严重降低模型在实际应用中的性能。最近的研究表明,造成性能下降的原因之一是一种过拟合现象,适当的正则化方法可以减轻性能下降的影响,尤其是使用高度代表性的模型,例如神经网络。本文提出了一种新的正则化方法,使用有监督对比学习方法来防止这种过拟合,训练出不会在分布漂移情况下降低性能的模型。我们扩展了对比损失中的余弦相似度到更一般的相似性度量,并提出在比较样本和正负例时使用不同的参数,这被理论上证明可以作为对比损失的一种边界。在模拟分布漂移的基准数据集上的实验,包括子群体变换和域泛化的实验,证明了提出的方法优于现有的正则化方法。