Spiking Neural Networks (SNNs) are biologically realistic and practically promising in low-power computation because of their event-driven mechanism. Usually, the training of SNNs suffers accuracy loss on various tasks, yielding an inferior performance compared with ANNs. A conversion scheme is proposed to obtain competitive accuracy by mapping trained ANNs' parameters to SNNs with the same structures. However, an enormous number of time steps are required for these converted SNNs, thus losing the energy-efficient benefit. Utilizing both the accuracy advantages of ANNs and the computing efficiency of SNNs, a novel SNN training framework is proposed, namely layer-wise ANN-to-SNN knowledge distillation (LaSNN). In order to achieve competitive accuracy and reduced inference latency, LaSNN transfers the learning from a well-trained ANN to a small SNN by distilling the knowledge other than converting the parameters of ANN. The information gap between heterogeneous ANN and SNN is bridged by introducing the attention scheme, the knowledge in an ANN is effectively compressed and then efficiently transferred by utilizing our layer-wise distillation paradigm. We conduct detailed experiments to demonstrate the effectiveness, efficacy, and scalability of LaSNN on three benchmark data sets (CIFAR-10, CIFAR-100, and Tiny ImageNet). We achieve competitive top-1 accuracy compared to ANNs and 20x faster inference than converted SNNs with similar performance. More importantly, LaSNN is dexterous and extensible that can be effortlessly developed for SNNs with different architectures/depths and input encoding methods, contributing to their potential development.
翻译:脉冲神经网络(SNNs)由于其事件驱动机制而在低功耗计算方面具有生物学上的现实性和实际的优势。通常,SNNs的训练在各种任务上都存在精度损失,与ANNs相比表现不佳。提出了一种转换方案,通过将训练过的ANNs参数映射到具有相同结构的SNNs上,获得具有竞争力的精度。然而,这些转换后的SNNs需要大量的时间步长,因此失去了节能的好处。利用ANNs的精度优势和SNNs的计算效率,提出了一种新的SNN训练框架,即逐层ANN到SNN知识蒸馏(LaSNN)。为了实现竞争性的精度和降低推理延迟,LaSNN通过蒸馏ANN其他知识而不是转换ANN参数,将学习从训练良好的ANN转移到小型SNN。通过引入注意力机制来弥合异质ANN和SNN之间的信息差距,使用我们的逐层蒸馏范式,可以有效地压缩ANN中的知识,然后进行高效的转移。我们通过在三个基准数据集(CIFAR-10,CIFAR-100和Tiny ImageNet)上进行详细实验来展示LaSNN的有效性,效率和可扩展性。与ANNs相比,我们实现了有竞争力的top-1精度,并且比性能相似的转换SNN的推理速度快20倍。更重要的是,LaSNN灵活和可扩展,可以轻松地针对具有不同体系结构/深度和输入编码方法的SNN进行开发,为其潜在发展做出贡献。