In this paper, we analyze the landscape of the true loss of one-hidden-layer networks with ReLU, leaky ReLU, as well as with quadratic activation. In all three cases, we provide a complete classification of the critical points in the case where the target function is affine. In particular, we show that there exist no local maxima and clarify the structure of saddle points. Moreover, we prove that non-global local minima can only be caused by `dead' ReLU neurons. In particular, they do not appear in the case of leaky ReLU or quadratic activation. Our approach is of a combinatorial nature and builds on a careful analysis of the different types of hidden neurons that can occur.
翻译:在本文中,我们分析了与ReLU、渗漏 ReLU以及二次激活的单层网络真正丢失的场景。 在这三个案例中,我们提供了目标功能为直线的案例中关键点的完整分类。特别是,我们显示当地没有最高标准,并澄清了马鞍点的结构。此外,我们还证明,非全球本地小型网络只能由“死”RELU神经元造成。特别是,它们并不出现在泄漏 ReLU或二次激活的情况下。我们的方法具有组合性质,并且建立在对可能发生的不同类型隐蔽神经元进行仔细分析的基础上。