Despite the great success of self-supervised learning with large floating point networks, such networks are not readily deployable to edge devices. To accelerate deployment of models to edge devices for various downstream tasks by unsupervised representation learning, we propose a self-supervised learning method for binary networks. In particular, we propose to use a randomly initialized classifier attached to a pretrained floating point feature extractor as targets and jointly train it with a binary network. For better training of the binary network, we propose a feature similarity loss, a dynamic balancing scheme of loss terms, and modified multi-stage training. We call our method as BSSL. Our empirical validations show that BSSL outperforms self-supervised learning baselines for binary networks in various downstream tasks and outperforms supervised pretraining in certain tasks.
翻译:尽管与大型浮点网络进行自我监督的学习取得了巨大成功,但这种网络并非随时可以部署到边缘装置。为了通过未经监督的代表学习加快将模型部署到边缘装置,以完成各种下游任务,我们提议为二进网采用自监督的学习方法。特别是,我们提议使用一个随机初始化的分类器作为目标,并用一个二进网对它进行联合培训。为了更好地培训二进网,我们建议采用一个特征相似性损失、一个动态平衡的损失条件计划和经过修改的多阶段培训。我们称我们的方法为BSSL。我们的经验验证表明,BSSL为各种下游任务中的二进网制定了自我监督的学习基线,并且在某些任务中超越监督的预培训。