Contrastive learning has been applied successfully to learn numerical vector representations of various forms of data, such as texts and images. Learned encoders exhibit versatile transfer capabilities to many downstream tasks. Representation based search is highly efficient with state-of-the-art performance. Previous researches demonstrated that learning high-quality representations requires a large number of negatives in contrastive loss. In practice, the technique of in-batch negative is used, where for each example in a batch, other batch examples' positives will be taken as its negatives, avoiding encoding extra negatives. This, however, still conditions each example's loss on all batch examples and requires fitting the entire large batch into GPU memory. This paper introduces a re-computation technique that decouples back propagation between contrastive loss and the encoder, removing encoder backward pass data dependency along the batch dimension. As a result, gradients can be computed for one subset of the batch at a time, leading to an almost constant peak GPU memory usage for batches of different sizes.
翻译:成功应用了对比性学习来学习多种数据形式(如文本和图像)的数字矢量表达方式,例如文本和图像。 进化编码器显示许多下游任务具有多种传导能力。 基于代表的搜索效率很高,与最先进的性能相当。 以前的研究表明,学习高质量表达方式需要大量的负值,而对比性损失则需要大量的负值。 在实践中,使用批量负值技术, 对于每批中的每一例来说,其他批次示例的正值将被当作其负值, 避免编码额外的负值。 然而, 这仍然使每个例的流失在所有批次实例中都成为条件, 需要将整批大批次都安装到 GPU 内存中。 本文引入了一种重新计算技术, 将对比性损失和编码器之间的扩散分批量消除编码落后数据依赖性。 结果, 一次可以计算出每批次组组次的梯度的梯度, 导致不同尺寸的批次的GPU记忆使用几乎稳定的峰值。