The pretrain-finetune paradigm is a classical pipeline in visual learning. Recent progress on unsupervised pretraining methods shows superior transfer performance to their supervised counterparts. This paper revisits this phenomenon and sheds new light on understanding the transferability gap between unsupervised and supervised pretraining from a multilayer perceptron (MLP) perspective. While previous works focus on the effectiveness of MLP on unsupervised image classification where pretraining and evaluation are conducted on the same dataset, we reveal that the MLP projector is also the key factor to better transferability of unsupervised pretraining methods than supervised pretraining methods. Based on this observation, we attempt to close the transferability gap between supervised and unsupervised pretraining by adding an MLP projector before the classifier in supervised pretraining. Our analysis indicates that the MLP projector can help retain intra-class variation of visual features, decrease the feature distribution distance between pretraining and evaluation datasets, and reduce feature redundancy. Extensive experiments on public benchmarks demonstrate that the added MLP projector significantly boosts the transferability of supervised pretraining, e.g. +7.2% top-1 accuracy on the concept generalization task, +5.8% top-1 accuracy for linear evaluation on 12-domain classification tasks, and +0.8% AP on COCO object detection task, making supervised pretraining comparable or even better than unsupervised pretraining.
翻译:在视觉学习中,先导-飞毛腿范式是一种典型的视觉学习管道。在未经监督的培训前方法方面最近的进展显示了向受监督的同行转移的优劣性能。本文件再次审视了这一现象,并从多层感官(MLP)的角度,为理解未经监督和监管的预培训之间在多层感官(MLP)角度的可转让性差距提供了新的见解。虽然以前的工作重点是MLP在未经监督的图像分类方面的有效性,在对同一数据集进行预先培训和评价的情况下,我们发现MLP投影机也是使未经监督的预培训方法比受监督的预培训方法更可转让性的关键因素。基于这一观察,我们试图通过在接受监督的预导师(MLP)之前添加一个MLP投影师来缩小受监督的可转让性差距。我们的分析表明,MLP投影师可以帮助保留在未监督的图像分类方面的内部差异,减少预培训和评价数据集之间的特征分布距离,并减少特征冗余性。关于公共基准的广泛实验表明,所增加的MLP投影师甚至提升了受监督的标前0的标值,在受监督的升级之前的准确性任务中,eLPLPOVI+1级前任务上,在高级任务上,在受监督的准确性任务上,在受监督的高级任务上,在受监督的升级前一级,在高级任务中提高了的不精确性任务中,在高级任务中,在高级任务上,在高级任务上,在高级任务上,在高级任务上,在高级任务上,在高级任务上,在高级任务上,在高级任务上,在高级任务上,在高级任务上,在高级任务上,在高级任务上,在高级任务上,在高级任务上,8+。