We consider the approximation rates of shallow neural networks with respect to the variation norm. Upper bounds on these rates have been established for sigmoidal and ReLU activation functions, but it has remained an important open problem whether these rates are sharp. In this article, we provide a solution to this problem by proving sharp lower bounds on the approximation rates for shallow neural networks, which are obtained by lower bounding the $L^2$-metric entropy of the convex hull of the neural network basis functions. In addition, our methods also give sharp lower bounds on the Kolmogorov $n$-widths of this convex hull, which show that the variation spaces corresponding to shallow neural networks cannot be efficiently approximated by linear methods. These lower bounds apply to both sigmoidal activation functions with bounded variation and to activation functions which are a power of the ReLU. Our results also quantify how much stronger the Barron spectral norm is than the variation norm and, combined with previous results, give the asymptotics of the $L^\infty$-metric entropy up to logarithmic factors in the case of the ReLU activation function.
翻译:我们考虑的是浅神经网络相对于变异规范的近似率。 这些光线网络与神经网络基本功能的光线质质质测试值的上限已经确定,但对于这些电流启动功能而言,这些率的上限仍然是一个重要的尚未解决的问题。在本篇文章中,我们通过证明浅神经网络的近似率的下限是明显较低的,而浅神经网络的近似率则是通过较低约束神经网络基本功能的锥形壳的$L$2当量的酶。此外,我们的方法还给这个 convex 壳体的 Kolmogorov $n$-width 设定了显著的下限,这表明与浅神经网络相对的变异空间无法有效地被线性方法所近似。这些下限既适用于带有受约束变异的模拟激活功能,也适用于作为ReLU的动力的激活功能。我们的结果还量化了Barron光谱规范比变异规范强多少,并且结合先前的结果,还给出了Repticropy 函数的Repticretical 系数。