We provide an entropy bound for the spaces of neural networks with piecewise linear activation functions, such as the ReLU and the absolute value functions. This bound generalizes the known entropy bound for the space of linear functions on $\mathbb{R}^d$ and it depends on the value at the point $(1,1,...,1)$ of the networks obtained by taking the absolute values of all parameters of original networks. Keeping this value together with the depth, width and the parameters of the networks to have logarithmic dependence on $1/\varepsilon$, we $\varepsilon$-approximate functions that are analytic on certain regions of $\mathbb{C}^d$. As a statistical application we derive an oracle inequality for the expected error of the considered penalized deep neural network estimators.
翻译:我们为神经网络空间提供了一条线性激活功能,如ReLU 和绝对值函数。 这个线性连接将已知线性函数空间线性函数的环球光化为$\mathbb{R ⁇ d$, 取决于通过使用原始网络所有参数的绝对值而获得的网络值(1, 1,...,..., 1)$。 将这一值与网络的深度、 宽度和参数结合起来, 对1美元/\ varepsilon$具有对数依赖性。 我们$\ varepsilon$- pappoint 函数对某区域 $\\ mathbb{C ⁇ d$ 。 作为统计应用程序,我们为被认为受罚的深神经网络估计值的预期错误得出了一个符号。