Most existing Spiking Neural Network (SNN) works state that SNNs may utilize temporal information dynamics of spikes. However, an explicit analysis of temporal information dynamics is still missing. In this paper, we ask several important questions for providing a fundamental understanding of SNNs: What are temporal information dynamics inside SNNs? How can we measure the temporal information dynamics? How do the temporal information dynamics affect the overall learning performance? To answer these questions, we estimate the Fisher Information of the weights to measure the distribution of temporal information during training in an empirical manner. Surprisingly, as training goes on, Fisher information starts to concentrate in the early timesteps. After training, we observe that information becomes highly concentrated in earlier few timesteps, a phenomenon we refer to as temporal information concentration. We observe that the temporal information concentration phenomenon is a common learning feature of SNNs by conducting extensive experiments on various configurations such as architecture, dataset, optimization strategy, time constant, and timesteps. Furthermore, to reveal how temporal information concentration affects the performance of SNNs, we design a loss function to change the trend of temporal information. We find that temporal information concentration is crucial to building a robust SNN but has little effect on classification accuracy. Finally, we propose an efficient iterative pruning method based on our observation on temporal information concentration. Code is available at https://github.com/Intelligent-Computing-Lab-Yale/Exploring-Temporal-Information-Dynamics-in-Spiking-Neural-Networks.
翻译:大多数现有的Spiking神经网络(SNNN)工作表明,SNNS可能使用时间信息动态的峰值。然而,对时间信息动态的清晰分析仍然缺乏。在本文中,我们问了几个重要问题,以提供对SNNs的基本理解:SNNs中什么是时间信息动态?我们如何测量时间信息动态?时间信息动态如何影响整个学习绩效?为了回答这些问题,我们用经验方式估算Fisher信息重量,以测量培训期间时间信息分布。令人惊讶的是,随着培训的进行,Fisher信息开始在早期时间步骤中集中。在培训之后,我们观察到信息高度集中在早期的几步中,我们称之为时间信息集中的现象。我们观察到,时间信息集中现象是SNNNIS的一个共同学习特征,通过对各种配置进行广泛的实验,例如结构、数据集、优化战略、时间恒定和时间步骤。此外,要揭示时间信息集中度如何影响SNNNR的绩效,我们设计一个丢失功能功能函数改变时间信息趋势。我们最后发现SNBR的时空浓度对S-rocent cent cental rogration roal-ral-ral-ral-ral-ral-al-lading to we fal-lading to lading to smakeding to