对比学习简述

基于自监督学习的Bert[1]预训练模型在NLP领域大放光彩，在多项下游任务中均取得很好的效果。Bert在无标注的语料中充分地学到了通用的知识，那么很容易引出一个问题，CV领域是否也可以“复现”Bert的成功呢？近年比较火热的对比学习或许是这个问题的一个答案。

对比学习（Contrastive Learning）是自监督学习的一种，需要从无标注的图像数据中学习特征表示，并用于下游任务中。其指导原则是：通过自动构造相似实例和不相似实例，学习一个表示学习模型，通过这个模型，使得相似的实例在投影空间中比较接近，而不相似的实例在投影空间中距离比较远。本文将介绍对比学习的基本思路以及经典的MoCo系列[2][3][4]、SimCLR系列模型[5][6]，了解对比学习的方法和特性。

[1] Devlin, Jacob, et al. "Bert: Pre-training of deep bidirectional transformers for language understanding." arXiv preprint arXiv:1810.04805 (2018).

[2] He, Kaiming, et al. "Momentum contrast for unsupervised visual representation learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

[3] Chen, Xinlei, et al. "Improved baselines with momentum contrastive learning." arXiv preprint arXiv:2003.04297 (2020).

[4] Chen, Xinlei, Saining Xie, and Kaiming He. "An empirical study of training self-supervised visual transformers." arXiv preprint arXiv:2104.02057 (2021).

[5] Chen, Ting, et al. "A simple framework for contrastive learning of visual representations." International conference on machine learning. PMLR, 2020.

[6] Chen, Ting, et al. "Big self-supervised models are strong semi-supervised learners." arXiv preprint arXiv:2006.10029 (2020).

[7] Contrastive Self-Supervised Learning https://ankeshanand.com/blog/2020/01/26/contrative-self-supervised-learning.html

[8] Kingma, Diederik P., and Max Welling. "Auto-encoding variational bayes." arXiv preprint arXiv:1312.6114 (2013).

[9] Goodfellow, Ian J., et al. "Generative adversarial networks." arXiv preprint arXiv:1406.2661 (2014).

[10] Caron, Mathilde, et al. "Unsupervised learning of visual features by contrasting cluster assignments." arXiv preprint arXiv:2006.09882 (2020).

[11] Grill, Jean-Bastien, et al. "Bootstrap your own latent: A new approach to self-supervised learning." arXiv preprint arXiv:2006.07733 (2020).

[12] Chen, Xinlei, and Kaiming He. "Exploring Simple Siamese Representation Learning." arXiv preprint arXiv:2011.10566 (2020).

[13] Gao, Tianyu, Xingcheng Yao, and Danqi Chen. "SimCSE: Simple Contrastive Learning of Sentence Embeddings." arXiv preprint arXiv:2104.08821 (2021).