Contrastive learning (CL) continuously achieves significant breakthroughs across multiple domains. However, the most common InfoNCE based methods suffer from some existing dilemmas, e.g., uniformity-tolerance dilemma (UTD) and the gradient reduction. It has been identified that UTD can lead to unexpected performance degradation. We argue that the fixity of temperature is to blame for UTD. To tackle this challenge, we enrich the CL loss family by presenting a Model-Aware Contrastive Learning (MACL) strategy, whose temperature is adaptive to the magnitude of alignment that reflects the basic confidence of the instance discrimination task, then enables CL loss to adjust the penalty strength for hard negatives adaptively. Regarding another dilemma, the gradient reduction issue, we derive the limits of an involved gradient scaling factor, which allows us to explain from a unified perspective why some recent approaches are effective with fewer negative samples, and summarily present a gradient reweighting to escape this dilemma. Extensive remarkable empirical results in vision, sentence, and graph modality validate our approach's general improvement for representation learning and downstream tasks.
翻译:反差学习(CL)在多个领域不断取得重大突破。然而,基于最常用的基于InfoNCE的方法存在一些现有的两难困境,例如统一容忍困境(UDD)和梯度降低。已经发现UTD可能导致意想不到的性能退化。我们认为,温度的固定性是UTD的罪魁祸首。为了应对这一挑战,我们提出了示范-意识反差学习(MACL)战略,丰富了CL损失家庭,其温度可适应反映歧视任务基本信心的调整幅度,然后使CL损失能够调整硬性负差的处罚力度。关于另一个两难困境,即梯度减少问题,我们从统一的角度来解释为什么最近的一些方法以较少的负数样本有效,并简要地展示了摆脱这一困境的梯度再加权。在愿景、句子和图表模式上的广泛显著的经验结果证实了我们的方法在代表性学习和下游任务方面的总体改进。