Spiking neural networks are becoming increasingly popular for their low energy requirement in real-world tasks with accuracy comparable to the traditional ANNs. SNN training algorithms face the loss of gradient information and non-differentiability due to the Heaviside function in minimizing the model loss over model parameters. To circumvent the problem surrogate method uses a differentiable approximation of the Heaviside in the backward pass, while the forward pass uses the Heaviside as the spiking function. We propose to use the zeroth order technique at the neuron level to resolve this dichotomy and use it within the automatic differentiation tool. As a result, we establish a theoretical connection between the proposed local zeroth-order technique and the existing surrogate methods and vice-versa. The proposed method naturally lends itself to energy-efficient training of SNNs on GPUs. Experimental results with neuromorphic datasets show that such implementation requires less than 1 percent neurons to be active in the backward pass, resulting in a 100x speed-up in the backward computation time. Our method offers better generalization compared to the state-of-the-art energy-efficient technique while maintaining similar efficiency.
翻译:Spik 神经网络因其在现实世界任务中的低能量需求而越来越受欢迎,其精确度与传统的ANNs相仿。 SNN培训算法面临梯度信息损失和无差别性的问题,因为Haviside功能在将模型损失与模型参数相比降低到最小方面。为了绕过问题代孕方法,在后转口使用一种不同的Heaviside近似值,而前转口使用Heaviside作为跳动功能。我们提议在神经层面使用零顺序技术来解决这一分化,并在自动分化工具中使用它。因此,我们在拟议的当地零顺序技术与现有的代管方法和反向法之间建立了理论联系。拟议的方法自然地有助于在GPU上对SNNS进行节能培训。神经变形数据集的实验结果显示,在后转口使用这种技术需要不到1%的神经元,从而在后转加速计算时间达到100x速度。我们的方法比州节能技术提供了更好的普及。