We explore the ability of overparameterized shallow neural networks to learn Lipschitz regression functions with and without label noise when trained by Gradient Descent (GD). To avoid the problem that in the presence of noisy labels, neural networks trained to nearly zero training error are inconsistent on this class, we propose an early stopping rule that allows us to show optimal rates. This provides an alternative to the result of Hu et al. (2021) who studied the performance of $\ell 2$ -regularized GD for training shallow networks in nonparametric regression which fully relied on the infinite-width network (Neural Tangent Kernel (NTK)) approximation. Here we present a simpler analysis which is based on a partitioning argument of the input space (as in the case of 1-nearest-neighbor rule) coupled with the fact that trained neural networks are smooth with respect to their inputs when trained by GD. In the noise-free case the proof does not rely on any kernelization and can be regarded as a finite-width result. In the case of label noise, by slightly modifying the proof, the noise is controlled using a technique of Yao, Rosasco, and Caponnetto (2007).
翻译:我们探讨过量的浅浅神经网络是否有能力在受Gradient Emproper(GD)培训时,学习使用和不带标签噪音的Lipschitz回归功能。为了避免在出现噪音标签的情况下,经过训练的神经网络在这个班级上几乎是零培训错误的问题不一致,我们提议了一项早期停止规则,使我们能够显示最佳比率。这为Hu等人(2021年)研究了2美元-正规化GD的性能,以培训完全依赖无限宽网(Neural Tangent Kernel(NTKKK)))近似的非临界回归的浅网络提供了一种替代方法。在这里,我们提出一项更简单的分析,其依据是对输入空间进行分割的争论(如1个近邻邻居规则),以及经过训练的神经网络在接受GD(2021年)培训时其投入方面是顺畅的。在无噪音的情况下,证据并不依赖任何内核化,而且可以被视为一种限定的边缘结果。在标签噪音的情况下,通过稍微修改证据,将噪音控制起来,并使用Casion 和Yasco (2007年) 技术。