Inspired by diversity of biological neurons, quadratic artificial neurons can play an important role in deep learning models. The type of quadratic neurons of our interest replaces the inner-product operation in the conventional neuron with a quadratic function. Despite promising results so far achieved by networks of quadratic neurons, there are important issues not well addressed. Theoretically, the superior expressivity of a quadratic network over either a conventional network or a conventional network via quadratic activation is not fully elucidated, which makes the use of quadratic networks not well grounded. Practically, although a quadratic network can be trained via generic backpropagation, it can be subject to a higher risk of collapse than the conventional counterpart. To address these issues, we first apply the spline theory and a measure from algebraic geometry to give two theorems that demonstrate better model expressivity of a quadratic network than the conventional counterpart with or without quadratic activation. Then, we propose an effective and efficient training strategy referred to as ReLinear to stabilize the training process of a quadratic network, thereby unleashing the full potential in its associated machine learning tasks. Comprehensive experiments on popular datasets are performed to support our findings and evaluate the performance of quadratic deep learning.
翻译:在生物神经多样性的启发下,四人造神经元在深层学习模型中可以发挥重要作用。我们感兴趣的四人神经元类型可以以二次函数取代常规神经元的内产品操作。尽管四人神经元网络迄今取得了有希望的成果,但也有一些重要问题没有很好解决。理论上,四人网络在常规网络或通过四人激活的常规网络或常规网络上的高度表达性没有得到充分阐明,这使得四人网络的使用没有很好的基础。实际上,尽管我们感兴趣的四人网络可以通过一般反向分析来培训,但它可能面临比常规对口网络更大的崩溃风险。为了解决这些问题,我们首先应用四人神经元网络的定线理论和测算尺度来给两种理论提供更好的模型表达性,这些理论显示四人网络比传统的对口网络和不使用四人激活的对口网络都更明显。然后,我们提出一个有效的培训战略,称为ReLin, 以稳定二次网络的培训进程,但这种网络的崩溃风险可能高于常规的对口网络。为了解决这些问题,我们首先应用定型理论理论和测算法的两种理论,以更好的模型来显示四人造网络的深层实验结果。我们进行的全面学习。