Model compression techniques are receiving increasing attention; however, the effect of compression on model fairness is still under explored. This is the first paper to examine the effect of distillation and pruning on the toxicity and bias of generative language models. We test Knowledge Distillation and Pruning methods on the GPT2 model and found a consistent pattern of toxicity and bias reduction after model distillation; this result can be potentially interpreted by existing line of research which describes model compression as a regularization technique; our work not only serves as a reference for safe deployment of compressed models, but also extends the discussion of "compression as regularization" into the setting of neural LMs, and hints at the possibility of using compression to develop fairer models.
翻译:模型压缩技术正受到越来越多的注意;然而,压缩对模型公平性的影响仍在探讨之中;这是审查蒸馏和裁剪对基因化语言模型的毒性和偏差的影响的第一份文件;我们在GPT2模型上测试知识蒸馏和预留方法,发现在模型蒸馏后毒性和偏差减少的一贯模式;这一结果有可能被现有研究线所解释,该研究线将模型压缩描述为一种正规化技术;我们的工作不仅作为安全部署压缩模型的参考,而且还将“作为正规化的压缩”的讨论扩展到神经液态模型的设置中,并暗示利用压缩开发更公平的模型的可能性。