Deep neural networks are parametrized by several thousands or millions of parameters, and have shown tremendous success in many classification problems. However, the large number of parameters makes it difficult to integrate these models into edge devices such as smartphones and wearable devices. To address this problem, knowledge distillation (KD) has been widely employed, that uses a pre-trained high capacity network to train a much smaller network, suitable for edge devices. In this paper, for the first time, we study the applicability and challenges of using KD for time-series data for wearable devices. Successful application of KD requires specific choices of data augmentation methods during training. However, it is not yet known if there exists a coherent strategy for choosing an augmentation approach during KD. In this paper, we report the results of a detailed study that compares and contrasts various common choices and some hybrid data augmentation strategies in KD based human activity analysis. Research in this area is often limited as there are not many comprehensive databases available in the public domain from wearable devices. Our study considers databases from small scale publicly available to one derived from a large scale interventional study into human activity and sedentary behavior. We find that the choice of data augmentation techniques during KD have a variable level of impact on end performance, and find that the optimal network choice as well as data augmentation strategies are specific to a dataset at hand. However, we also conclude with a general set of recommendations that can provide a strong baseline performance across databases.
翻译:深心神经网络被数千个或数百万个参数截断,并在许多分类问题中表现出巨大的成功。然而,由于参数众多,很难将这些模型纳入智能手机和磨损装置等边缘装置。为了解决这个问题,知识蒸馏(KD)被广泛使用,使用经过预先训练的高能力网络来训练一个更小、更适合边缘装置的网络。在本文件中,我们首次研究将KD用于可磨损装置的时间序列数据的可适用性和挑战。成功应用KD需要在培训中具体选择数据增强方法。然而,尚不清楚在选择KD期间的增强方法方面是否存在一致的战略。在本文中,我们报告了一项详细研究的结果,该研究比较并对比了各种共同选择以及基于KD的人类活动分析中的一些混合数据增强战略。由于在公共领域从可磨损装置中找不到许多全面数据库,因此这方面的研究往往有限。我们的研究认为,从小规模的数据库可以公开获得,而从大规模干预研究得出的数据扩增数据扩增人类活动及可变化的网络。我们发现,在一般的基线数据库中,我们发现,在特定数据分析阶段发现,我们可选择了一种特定的绩效,在特定数据层次上可以提供。