This paper introduces Relative Predictive Coding (RPC), a new contrastive representation learning objective that maintains a good balance among training stability, minibatch size sensitivity, and downstream task performance. The key to the success of RPC is two-fold. First, RPC introduces the relative parameters to regularize the objective for boundedness and low variance. Second, RPC contains no logarithm and exponential score functions, which are the main cause of training instability in prior contrastive objectives. We empirically verify the effectiveness of RPC on benchmark vision and speech self-supervised learning tasks. Lastly, we relate RPC with mutual information (MI) estimation, showing RPC can be used to estimate MI with low variance.
翻译:本文件介绍相对预测编码(RPC),这是一个新的对比性代表性学习目标,在培训稳定性、小批量尺寸敏感度和下游任务业绩之间保持良好平衡,这是成功实现RPC的关键是双重的。首先,RPC引入了相关参数,以规范约束性和低差异的目标。第二,RPC不包含对数和指数分数功能,这些功能是先前对比性目标培训不稳定的主要原因。我们实证了RPC在基准愿景和语言自我监督的学习任务方面的有效性。最后,我们将RPC与相互信息(MI)估算联系起来,表明RPC可用于低差异估算MI。