The implicit bias of neural networks has been extensively studied in recent years. Lyu and Li [2019] showed that in homogeneous networks trained with the exponential or the logistic loss, gradient flow converges to a KKT point of the max margin problem in the parameter space. However, that leaves open the question of whether this point will generally be an actual optimum of the max margin problem. In this paper, we study this question in detail, for several neural network architectures involving linear and ReLU activations. Perhaps surprisingly, we show that in many cases, the KKT point is not even a local optimum of the max margin problem. On the flip side, we identify multiple settings where a local or global optimum can be guaranteed.
翻译:近年来,对神经网络的隐含偏差进行了广泛研究。 Lyu和Li[2019]表明,在经过指数或后勤损失培训的同质网络中,梯度流会汇合到参数空间最大差幅问题的一个KKT点,然而,这就留下了这样一个问题,即这个点是否一般是最大差幅问题的一个实际最佳问题。在本文中,我们详细研究涉及线性激活和RELU激活的若干神经网络结构的这一问题。也许令人惊讶的是,我们发现,在许多情况下,KKT点甚至不是最大差幅问题的最佳地方。在反面,我们找出了可以保证地方或全球最佳地方或全球最佳的多种环境。