Data-free knowledge distillation is a challenging model lightweight task for scenarios in which the original dataset is not available. Previous methods require a lot of extra computational costs to update one or more generators and their naive imitate-learning lead to lower distillation efficiency. Based on these observations, we first propose an efficient unlabeled sample selection method to replace high computational generators and focus on improving the training efficiency of the selected samples. Then, a class-dropping mechanism is designed to suppress the label noise caused by the data domain shifts. Finally, we propose a distillation method that incorporates explicit features and implicit structured relations to improve the effect of distillation. Experimental results show that our method can quickly converge and obtain higher accuracy than other state-of-the-art methods.
翻译:无数据知识蒸馏对于原始数据集无法提供的假想情况来说是一项具有挑战性的模型轻重任务。 以往的方法需要大量额外的计算成本来更新一个或多个发电机,而它们的天真模仿学习可以降低蒸馏效率。 基于这些观察,我们首先提出一种高效的无标签样本选择方法,以取代高计算生成器,并侧重于提高选定样本的培训效率。 然后,一个分类滴记机制旨在抑制数据域变换造成的标签噪音。 最后,我们建议一种含有明确特征和隐含结构关系的蒸馏方法,以改进蒸馏效果。 实验结果显示,我们的方法可以很快趋同并获得比其他最先进的方法更高的精度。