We explore the use of knowledge distillation (KD) for learning compact and accurate models that enable classification of animal behavior from accelerometry data on wearable devices. To this end, we take a deep and complex convolutional neural network, known as residual neural network (ResNet), as the teacher model. ResNet is specifically designed for multivariate time-series classification. We use ResNet to distill the knowledge of animal behavior classification datasets into soft labels, which consist of the predicted pseudo-probabilities of every class for each datapoint. We then use the soft labels to train our significantly less complex student models, which are based on the gated recurrent unit (GRU) and multilayer perceptron (MLP). The evaluation results using two real-world animal behavior classification datasets show that the classification accuracy of the student GRU-MLP models improves appreciably through KD, approaching that of the teacher ResNet model. To further reduce the computational and memory requirements of performing inference using the student models trained via KD, we utilize dynamic fixed-point quantization (DQ) through an appropriate modification of the computational graph of the considered models. We implement both unquantized and quantized versions of the developed KD-based models on the embedded systems of our purpose-built collar and ear tag devices to classify animal behavior in situ and in real time. Our evaluations corroborate the effectiveness of KD and DQ in improving the accuracy and efficiency of in-situ animal behavior classification.
翻译:我们探索利用知识蒸馏法(KD)来学习精确的缩压模型(KD),以便从可磨损设备上的侵蚀测量数据中对动物行为进行分类。为此,我们使用一个深而复杂的进化神经网络,称为残余神经网络(ResNet),作为教师模型。ResNet是专门设计用于多变时间序列分类的。我们使用ResNet将动物行为分类数据集的知识提炼成软标签,该标签包括每个数据点每类的预测假概率。我们然后使用软标签来培训我们远不那么复杂的学生模型,这些模型以GRU(GRU)和多层透视器(MLP)为基础。使用两个真实世界动物行为分类数据集的评估结果表明,学生GRU-MLP模型的分类准确性通过KD,接近以教师ResNet为基础的模型。为了进一步减少使用通过KD培训的学生分类模型进行推断的计算和记忆要求,我们用动态的固定数据模型和KQ(D)的静态定型定型计算,通过适当修改我们公司定置的定置的定时和定型定型的定型定型的定型标签,从而在KQ计算中进行我们的定型的动物行为。