Recent studies on compression of pretrained language models (e.g., BERT) usually use preserved accuracy as the metric for evaluation. In this paper, we propose two new metrics, label loyalty and probability loyalty that measure how closely a compressed model (i.e., student) mimics the original model (i.e., teacher). We also explore the effect of compression with regard to robustness under adversarial attacks. We benchmark quantization, pruning, knowledge distillation and progressive module replacing with loyalty and robustness. By combining multiple compression techniques, we provide a practical strategy to achieve better accuracy, loyalty and robustness.
翻译:最近关于压缩未受过训练的语言模型的研究(例如BERT)通常使用保存的准确性作为评价的衡量标准。在本文件中,我们提出了两个新的衡量标准,即标签忠诚度和概率忠诚度,以测量压缩模型(即学生)与原始模型(即教师)的相似程度。我们还探讨了压缩在对抗性攻击中对稳健性的影响。我们以量化、修剪、知识蒸馏和渐进式模块取代忠诚性和稳健性为基准。我们结合多种压缩技术,提供了实现更准确性、忠诚性和稳健性的实用战略。