Contrastive representation learning seeks to acquire useful representations by estimating the shared information between multiple views of data. Here, the choice of data augmentation is sensitive to the quality of learned representations: as harder the data augmentations are applied, the views share more task-relevant information, but also task-irrelevant one that can hinder the generalization capability of representation. Motivated by this, we present a new robust contrastive learning scheme, coined R\'enyiCL, which can effectively manage harder augmentations by utilizing R\'enyi divergence. Our method is built upon the variational lower bound of R\'enyi divergence, but a na\"ive usage of a variational method is impractical due to the large variance. To tackle this challenge, we propose a novel contrastive objective that conducts variational estimation of a skew R\'enyi divergence and provide a theoretical guarantee on how variational estimation of skew divergence leads to stable training. We show that R\'enyi contrastive learning objectives perform innate hard negative sampling and easy positive sampling simultaneously so that it can selectively learn useful features and ignore nuisance features. Through experiments on ImageNet, we show that R\'enyi contrastive learning with stronger augmentations outperforms other self-supervised methods without extra regularization or computational overhead. Moreover, we also validate our method on other domains such as graph and tabular, showing empirical gain over other contrastive methods.
翻译:对比性代表性学习试图通过估算数据多角度差异之间的共享信息来获得有用的表达方式。 在这里, 数据增强选择对所学表达方式的质量敏感: 随着数据增强的难度加大, 观点共享了更多任务相关信息, 但也分享了可能阻碍代表性普遍化能力的任务相关信息。 受此驱动, 我们提出了一个新的强势对比性学习计划, 创建了R\'enyico, 利用 R\' enyi 差异可以有效地管理更硬的增强。 我们的方法建立在R\' enyi 差异的变异性较低约束上, 但是由于差异很大, 使用差异性方法的反向使用是不切实际的。 为了应对这一挑战, 我们提出了一个新的对比性目标, 进行差异性估算, 并提供一个理论保证, 利用差异性差异性估算如何导致稳定的培训。 我们显示, R\\ eny 对比性对比性学习目标具有内含的硬性负面抽样和简单正面抽样, 从而可以选择学习有用性特征, 忽略不易读取的特性, 因为差异性特征, 因为差异很大差异性的方法是巨大的差异性, 我们通过在图像升级的升级的机变化的升级的机变化的模型中, 我们学习了其他的方法, 也展示了其他的自我升级的自我升级的升级的升级的升级的计算方法。