State-of-the-art artificial neural networks (ANNs) require labelled data or feedback between layers, are often biologically implausible, and are vulnerable to adversarial attacks that humans are not susceptible to. On the other hand, Hebbian learning in winner-take-all (WTA) networks, is unsupervised, feed-forward, and biologically plausible. However, a modern objective optimization theory for WTA networks has been missing, except under very limiting assumptions. Here we derive formally such a theory, based on biologically plausible but generic ANN elements. Through Hebbian learning, network parameters maintain a Bayesian generative model of the data. There is no supervisory loss function, but the network does minimize cross-entropy between its activations and the input distribution. The key is a "soft" WTA where there is no absolute "hard" winner neuron, and a specific type of Hebbian-like plasticity of weights and biases. We confirm our theory in practice, where, in handwritten digit (MNIST) recognition, our Hebbian algorithm, SoftHebb, minimizes cross-entropy without having access to it, and outperforms the more frequently used, hard-WTA-based method. Strikingly, it even outperforms supervised end-to-end backpropagation, under certain conditions. Specifically, in a two-layered network, SoftHebb outperforms backpropagation when the training dataset is only presented once, when the testing data is noisy, and under gradient-based adversarial attacks. Notably, adversarial attacks that confuse SoftHebb are also confusing to the human eye. Finally, the model can generate interpolations of objects from its input distribution. All in all, SoftHebb extends Hebbian WTA theory with modern machine learning tools, thus making these networks relevant to pertinent issues in deep learning.
翻译:最新人造神经网络(ANNS) 需要贴贴标签的数据或层间反馈, 通常在生物上不可信, 并且容易受到人类无法接触到的对抗性攻击。 另一方面, Hebbian 在赢者- 通吃( WTA) 网络中学习, 不受监督、 反馈前和生物上合理。 然而, WTA 网络缺少现代目标优化理论, 但限制性假设除外。 我们在这里正式地根据生物上看似似似是但通用的 ANNE 元素来得出这样的理论。 通过 Hebbian 学习, 网络的参数维持了Bayesian 数据基因化模型模型模型模型。 没有监督性损失功能, 但是网络在启动和输入输入之间, Hebbian 将交叉oproopy- translation( WTA) 最小化。 关键是“ 软性” WTATA, 其中没有绝对的“ 硬性” 赢者神经, 以及特定类型的 Hebbbian 重度和偏差的塑料问题。 我们确认了我们的理论, 在实践中, 在手写数字中, 也可以化的数字中, 也可以化的数字中, 也可以化的理论中, 也可以化的理论中, 也可以在它在它的输入中, 和直读取数据传输中, 在它的输入中, 在它的输入中, 直向内, 在它的输入中, 数据流数据流中,, 在它的输入中,, 在它的输入中, 在它的输入中, 在它的输入中, 直径流数据流数据流数据流中, 。