Deep neural networks can suffer from the exploding and vanishing activation problem, in which the networks fail to train properly because the neural signals either amplify or attenuate across the layers and become saturated. While other normalization methods aim to fix the stated problem, most of them have inference speed penalties in those applications that require running averages of the neural activations. Here we extend the unitary framework based on Lie algebra to neural networks of any dimensionalities, overcoming the major constraints of the prior arts that limit synaptic weights to be square matrices. Our proposed unitary convolutional neural networks deliver up to 32% faster inference speeds and up to 50% reduction in permanent hard disk space while maintaining competitive prediction accuracy.
翻译:深神经网络可能会受到爆炸和消失的激活问题的影响, 网络在其中未能进行适当的培训, 因为神经信号在层层之间放大或减弱, 并变得饱和。 虽然其他正常化方法旨在解决上述问题, 但大部分在那些需要神经激活运行平均速度的应用程序中都有推断速度罚则。 我们在这里将基于Lie代数的单一框架扩大到任何维度的神经网络, 从而克服了将合成重量限制为平方基体的先前艺术的主要制约。 我们提议的单一共振神经网络在保持有竞争力的预测准确性的同时,可以达到32%的快速推论速度, 并将永久硬盘空间减少高达50% 。