As a successful approach to self-supervised learning, contrastive learning aims to learn invariant information shared among distortions of the input sample. While contrastive learning has yielded continuous advancements in sampling strategy and architecture design, it still remains two persistent defects: the interference of task-irrelevant information and sample inefficiency, which are related to the recurring existence of trivial constant solutions. From the perspective of dimensional analysis, we find out that the dimensional redundancy and dimensional confounder are the intrinsic issues behind the phenomena, and provide experimental evidence to support our viewpoint. We further propose a simple yet effective approach MetaMask, short for the dimensional Mask learned by Meta-learning, to learn representations against dimensional redundancy and confounder. MetaMask adopts the redundancy-reduction technique to tackle the dimensional redundancy issue and innovatively introduces a dimensional mask to reduce the gradient effects of specific dimensions containing the confounder, which is trained by employing a meta-learning paradigm with the objective of improving the performance of masked representations on a typical self-supervised task. We provide solid theoretical analyses to prove MetaMask can obtain tighter risk bounds for downstream classification compared to typical contrastive methods. Empirically, our method achieves state-of-the-art performance on various benchmarks.
翻译:作为自我监督学习的成功方法,对比式学习旨在学习投入抽样扭曲之间共享的不固定信息。对比式学习在抽样战略和结构设计方面不断取得进步,但仍有两个持续存在的缺陷:与任务相关的信息的干扰和抽样效率低下,这与反复存在的琐碎不变的常态解决办法有关。从维学分析的角度来看,我们发现,多元冗余和维分分解者是现象背后的内在问题,并提供了实验性证据来支持我们的观点。我们进一步建议一种简单而有效的方法Metamask,这是Metamask所学的维维面面具的短处,用来学习对抗维度冗余冗余和复合器的演示。Metamask采用裁量技术来解决维度冗余问题,并以创新方式引入一个维面面面面面面面面面罩,以减少构成共体的具体维度的梯度的梯度效应,通过采用元学习模式进行培训,目的是改进典型自我监督任务蒙蔽的表达的性能。我们提供了坚实的理论分析,以证明MetMask在下游等级上可以得到更紧密的风险约束,与典型的对比性业绩方法相比较。