Lateral inhibitory connections have been observed in the cortex of the biological brain, and has been extensively studied in terms of its role in cognitive functions. However, in the vanilla version of backpropagation in deep learning, all gradients (which can be understood to comprise of both signal and noise gradients) flow through the network during weight updates. This may lead to overfitting. In this work, inspired by biological lateral inhibition, we propose Gradient Mask, which effectively filters out noise gradients in the process of backpropagation. This allows the learned feature information to be more intensively stored in the network while filtering out noisy or unimportant features. Furthermore, we demonstrate analytically how lateral inhibition in artificial neural networks improves the quality of propagated gradients. A new criterion for gradient quality is proposed which can be used as a measure during training of various convolutional neural networks (CNNs). Finally, we conduct several different experiments to study how Gradient Mask improves the performance of the network both quantitatively and qualitatively. Quantitatively, accuracy in the original CNN architecture, accuracy after pruning, and accuracy after adversarial attacks have shown improvements. Qualitatively, the CNN trained using Gradient Mask has developed saliency maps that focus primarily on the object of interest, which is useful for data augmentation and network interpretability.
翻译:在生物大脑皮层中观察到了横向抑制性联系,并广泛研究了生物大脑认知功能的作用。然而,在深层学习的香草版反反反演中,所有梯度(可以理解为包括信号和噪声梯度)在重量更新期间通过网络流动。这可能导致过度匹配。在生物横向抑制的启发下,我们提议使用梯度面具,在这项工作中,在反向调整过程中有效地过滤噪音梯度。这样可以使学到的特征信息在网络中更加密集地储存,同时过滤噪音或不重要的特征。此外,我们从分析角度展示了人工神经网络的横向抑制如何提高传播梯度的质量。提出了一个新的梯度质量标准,可以在各种进化神经网络(CNNs)培训过程中用作衡量标准。最后,我们进行了一些不同的实验,研究梯度面具如何在定量和定性上改进网络的性能。在原始CNN目标结构中,原始目标的准确性,在经过精准性神经攻击之后的准确性精确性,在经过精准性研究后,对准性攻击的精确性分析后,对质性攻击作了精确性研究,对准性研究后,对准性攻击作了精确性研究后,对准性攻击作了精确性研究。