High-Order Approximation Rates for Neural Networks with ReLU$^k$ Activation Functions

We study the approximation properties of shallow neural networks (NN) with activation function which is a power of the rectified linear unit. Specifically, we consider the dependence of the approximation rate on the dimension and the smoothness of the underlying function to be approximated. Like the finite element method, such networks represent piecewise polynomial functions. However, we show that for sufficiently smooth functions the approximation properties of shallow ReLU$^k$ networks are much better than finite elements or wavelets, and they even overcome the curse of dimensionality more effectively than the sparse grid method. Specifically, for a sufficiently smooth function $f$, there exists a ReLU$^k$-NN with $n$ neurons which approximates $f$ in $L^2([0,1]^d)$ with $O(n^{-(k+1)}\log(n))$ error. Finally, we prove lower bounds showing that the approximation rates attained are optimal under the given assumptions.