In this paper, we analyze the landscape of the true loss of neural networks with one hidden layer and ReLU, leaky ReLU, or quadratic activation. In all three cases, we provide a complete classification of the critical points in the case where the target function is affine and one-dimensional. In particular, we show that there exist no local maxima and clarify the structure of saddle points. Moreover, we prove that non-global local minima can only be caused by `dead' ReLU neurons. In particular, they do not appear in the case of leaky ReLU or quadratic activation. Our approach is of a combinatorial nature and builds on a careful analysis of the different types of hidden neurons that can occur.
翻译:在本文中,我们分析了一个隐藏层和ReLU、泄漏 ReLU或二次激活的神经网络真正丧失的全貌。 在这三个案例中,我们提供了目标功能为直线和一维的临界点的完整分类。特别是,我们显示没有本地的峰值,并澄清了马鞍点的结构。此外,我们证明,非全球的当地微型网络只能由`死'RELU神经元造成。特别是,它们并不出现在泄漏 ReLU或二次激活的情况下。我们的方法具有组合性质,并且建立在对可能发生的不同类型隐藏神经元的仔细分析的基础上。