With the success of self-supervised learning (SSL), it has become a mainstream paradigm to fine-tune from self-supervised pretrained models to boost the performance on downstream tasks. However, we find that current SSL models suffer severe accuracy drops when performing low-bit quantization, prohibiting their deployment in resource-constrained applications. In this paper, we propose a method called synergistic self-supervised and quantization learning (SSQL) to pretrain quantization-friendly self-supervised models facilitating downstream deployment. SSQL contrasts the features of the quantized and full precision models in a self-supervised fashion, where the bit-width for the quantized model is randomly selected in each step. SSQL not only significantly improves the accuracy when quantized to lower bit-widths, but also boosts the accuracy of full precision models in most cases. By only training once, SSQL can then benefit various downstream tasks at different bit-widths simultaneously. Moreover, the bit-width flexibility is achieved without additional storage overhead, requiring only one copy of weights during training and inference. We theoretically analyze the optimization process of SSQL, and conduct exhaustive experiments on various benchmarks to further demonstrate the effectiveness of our method. Our code is available at https://github.com/megvii-research/SSQL-ECCV2022.
翻译:随着自我监督学习的成功(SSL),它已经成为主流模式,从自我监督的预先训练模型中微调自监督的自我监督型号,以提高下游任务的业绩。然而,我们发现,当前SSL模型在进行低位四分制时会受到严重精度下降的影响,禁止将其部署于资源受限制的应用中。在本文件中,我们提出了一个称为协同自监督和量化学习的方法(SSQL),用于为下游部署提供便利。SSQL以自我监督的方式将量化和完全精准型号的功能与自我监督型号的特征作对比,在每步中随机选择四分化型模型的比重下降。SSQL不仅显著提高了在微调到低位四分分立的应用程序中的准确度,而且提高了多数情况下完全精确模型的准确性。只有一次培训,SSQL能够同时为不同位三分立的下游任务带来好处。此外,在不需额外存储的存储式二分解模式中,在SAR-LBS-L标准中实现的比重灵活度灵活性,只需要对我们的深度模型的精确度进行一次分析。