On-device Deep Neural Networks (DNNs) have recently gained more attention due to the increasing computing power of the mobile devices and the number of applications in Computer Vision (CV), Natural Language Processing (NLP), and Internet of Things (IoTs). Unfortunately, the existing efficient convolutional neural network (CNN) architectures designed for CV tasks are not directly applicable to NLP tasks and the tiny Recurrent Neural Network (RNN) architectures have been designed primarily for IoT applications. In NLP applications, although model compression has seen initial success in on-device text classification, there are at least three major challenges yet to be addressed: adversarial robustness, explainability, and personalization. Here we attempt to tackle these challenges by designing a new training scheme for model compression and adversarial robustness, including the optimization of an explainable feature mapping objective, a knowledge distillation objective, and an adversarially robustness objective. The resulting compressed model is personalized using on-device private training data via fine-tuning. We perform extensive experiments to compare our approach with both compact RNN (e.g., FastGRNN) and compressed RNN (e.g., PRADO) architectures in both natural and adversarial NLP test settings.
翻译:最近,由于移动设备的计算能力和计算机视野(CV)、自然语言处理(NLP)和物联网(IoTs)应用量的增加,用于CV任务的现有高效进化神经网络(CNN)结构不直接适用于NLP任务,而小型的经常性神经网络(RNN)结构主要是为IoT应用设计的。在NLP应用中,尽管模型压缩在对视文本分类方面初步取得了成功,但至少还有三大挑战有待解决:对抗性强、解释性和个人化。我们试图通过设计新的模型压缩和对抗性强力培训计划来应对这些挑战,包括优化可解释的特征绘图目标、知识蒸馏目标和对抗性强性目标。由此形成的压缩模型通过微调利用在线私人培训数据进行个性化化。我们进行了广泛的实验,以将我们的方法与NNNW、RNP、RGNF、RNF和RGF测试环境(RGR)的压缩机、RGNM、RGR、RNF、RGNR、RM、RGNF、RGN、RRG、RRRRRR、RRG、RGR、RGN、RRRG、RG、RG、RRG、RGRRRR、RRR、R、R、R、RR、I)两个的测试)的常规和RF等的常规和RF等的常规结构都都都。