Spiking Neural Networks (SNNs) have recently emerged as an alternative to deep learning owing to sparse, asynchronous and binary event (or spike) driven processing, that can yield huge energy efficiency benefits on neuromorphic hardware. However, training high-accuracy and low-latency SNNs from scratch suffers from non-differentiable nature of a spiking neuron. To address this training issue in SNNs, we revisit batch normalization and propose a temporal Batch Normalization Through Time (BNTT) technique. Most prior SNN works till now have disregarded batch normalization deeming it ineffective for training temporal SNNs. Different from previous works, our proposed BNTT decouples the parameters in a BNTT layer along the time axis to capture the temporal dynamics of spikes. The temporally evolving learnable parameters in BNTT allow a neuron to control its spike rate through different time-steps, enabling low-latency and low-energy training from scratch. We conduct experiments on CIFAR-10, CIFAR-100, Tiny-ImageNet and event-driven DVS-CIFAR10 datasets. BNTT allows us to train deep SNN architectures from scratch, for the first time, on complex datasets with just few 25-30 time-steps. We also propose an early exit algorithm using the distribution of parameters in BNTT to reduce the latency at inference, that further improves the energy-efficiency.
翻译:最近,由于零散的、不同步的和二进制的加工(或冲刺的)工艺,对神经变异硬件可产生巨大的能效效益,因此作为深层学习的替代方法,对高精度和低长的神经网络进行了从零到零的训练,但对高精度和低长的 SNNN 进行从零到零的培训,其性质是不可区分的。为了解决SNNN的这一培训问题,我们重新审视批次正常化,并提出从零到零的暂时批次正常化(BNTT)技术。迄今为止,SNNNNN的多数工作忽视了将它视为对时间性SNNNN的训练无效的分批正常化。不同于以前的工作,我们提议的BNTTT在BNTT的一层中将参数分解成巨大的节能效益。BNTTT在时间轴上逐渐变化的参数使得神经能通过不同的时间步骤控制其涨幅速度,从零到低能训练。我们先对CFAR-100、小点网络和事件变本-事件变本的参数进行试验,在SNTTFAR的最初的S-RO-RO-RO-RO-RO-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-TFAR-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S-S