Minimizing cross-entropy is a widely used method for training artificial neural networks. Many training procedures based on backpropagation use cross-entropy directly as their loss function. Instead, this theoretical essay investigates a dual process model with two processes, in which one process minimizes the Kullback-Leibler divergence while its dual counterpart minimizes the Shannon entropy. Postulating that learning consists of two dual processes complementing each other, the model defines an equilibrium state for both processes in which the loss function assumes its minimum. An advantage of the proposed model is that it allows deriving the optimal learning rate and momentum weight to update network weights for backpropagation. Furthermore, the model introduces the golden ratio and complex numbers as important new concepts in machine learning.
翻译:尽量减少交叉孔径是培训人工神经网络的一种广泛使用的方法。许多基于后向传播的培训程序直接使用交叉孔径作为损失功能。相反,本理论论文对两种过程的双重过程模式进行了调查,其中一种过程最大限度地缩小了Kullback-Lebeler的差异,而另一种过程的双重对手则尽量减少了香农的恒温。假设学习包括两个相辅相成的双重过程,该模型为两种过程确定了一种平衡状态,两种过程的损失功能都承担了最低限度。拟议模式的一个优点是,它能够得出最佳学习率和动力重量,以更新网络重量,用于后向传播。此外,该模型还引入黄金比率和复杂数字,作为机器学习中的重要新概念。