Sparse neural networks attract increasing interest as they exhibit comparable performance to their dense counterparts while being computationally efficient. Pruning the dense neural networks is among the most widely used methods to obtain a sparse neural network. Driven by the high training cost of such methods that can be unaffordable for a low-resource device, training sparse neural networks sparsely from scratch has recently gained attention. However, existing sparse training algorithms suffer from various issues, including poor performance in high sparsity scenarios, computing dense gradient information during training, or pure random topology search. In this paper, inspired by the evolution of the biological brain and the Hebbian learning theory, we present a new sparse training approach that evolves sparse neural networks according to the behavior of neurons in the network. Concretely, by exploiting the cosine similarity metric to measure the importance of the connections, our proposed method, Cosine similarity-based and Random Topology Exploration (CTRE), evolves the topology of sparse neural networks by adding the most important connections to the network without calculating dense gradient in the backward. We carried out different experiments on eight datasets, including tabular, image, and text datasets, and demonstrate that our proposed method outperforms several state-of-the-art sparse training algorithms in extremely sparse neural networks by a large gap. The implementation code is available on https://github.com/zahraatashgahi/CTRE
翻译:松散的神经网络在计算效率的同时,表现出与密集的神经网络相似的性能,因此吸引了越来越多的兴趣。 淡化稠密的神经网络是用来获取稀有神经网络的最广泛使用的方法之一。 受这种对低资源设备来说负担不起的方法的高培训成本的驱使,培训稀散的神经网络最近引起了人们的注意。 然而,现有的稀疏的培训算法存在各种问题,包括高空间情景的性能不佳,培训期间计算密度的梯度信息,或纯粹随机的地形搜索。 本文在生物大脑和赫比亚学习理论演变的启发下,提出了一种新的稀薄的培训方法,根据网络神经人的行为发展稀薄的神经网络。 具体地说,通过利用同源相似的量度测量连接的重要性,我们拟议的方法,科斯廷相似性和随机地形探索(CTREE),通过在不计算落后的密度梯度梯度的情况下,将稀薄神经网络的表面学进化为最重要的连接。 我们在8个数据库上进行了不同的实验, 包括表层式、 稀有的算法, 展示了我们现有的大量结构图象 。