Recently, sparse training methods have started to be established as a de facto approach for training and inference efficiency in artificial neural networks. Yet, this efficiency is just in theory. In practice, everyone uses a binary mask to simulate sparsity since the typical deep learning software and hardware are optimized for dense matrix operations. In this paper, we take an orthogonal approach, and we show that we can train truly sparse neural networks to harvest their full potential. To achieve this goal, we introduce three novel contributions, specially designed for sparse neural networks: (1) a parallel training algorithm and its corresponding sparse implementation from scratch, (2) an activation function with non-trainable parameters to favour the gradient flow, and (3) a hidden neurons importance metric to eliminate redundancies. All in one, we are able to break the record and to train the largest neural network ever trained in terms of representational power -- reaching the bat brain size. The results show that our approach has state-of-the-art performance while opening the path for an environmentally friendly artificial intelligence era.
翻译:最近,在人工神经网络中,开始将稀有的培训方法确定为事实上的培训和推断效率方法。然而,这种效率只是理论上的。在实践上,每个人都使用二进制面具模拟宽度,因为典型的深层学习软件和硬件被优化用于密集的矩阵操作。在本文中,我们采取正统方法,我们表明我们可以培训真正稀疏的神经网络,以充分发挥其潜力。为了实现这一目标,我们引入了三种新颖的贡献,专门为稀薄的神经网络设计:(1) 平行的培训算法及其从零开始的相应稀有实施;(2) 一种带有非可控制参数的激活功能,以有利于梯度流动;(3) 一种隐性神经元的重要性度,以消除冗余性。在本文中,我们可以打破记录,培训有史以来在代表性能力方面受过培训的最大神经网络 -- -- 达到蝙蝠脑规模。结果显示,我们的方法具有最先进的性能,同时开辟了环境友好人工智能时代的道路。