We focus on learning a two-layer residual neural network with preactivation by ReLU (preReLU-TLRN): Suppose the input $\mathbf{x}$ is from a distribution with support space $\mathbb{R}^d$ and the ground-truth generative model is a preReLU-TLRN, given by $$\mathbf{y} = \boldsymbol{B}^\ast\left[\left(\boldsymbol{A}^\ast\mathbf{x}\right)^+ + \mathbf{x}\right]\text{,}$$ where ground-truth network parameters $\boldsymbol{A}^\ast \in \mathbb{R}^{d\times d}$ is a nonnegative full-rank matrix and $\boldsymbol{B}^\ast \in \mathbb{R}^{m\times d}$ is full-rank with $m \geq d$. We design layerwise objectives as functionals whose analytic minimizers sufficiently express the exact ground-truth network in terms of its parameters and nonlinearities. Following this objective landscape, learning a preReLU-TLRN from finite samples can be formulated as convex programming with nonparametric function estimation: For each layer, we first formulate the corresponding empirical risk minimization (ERM) as convex quadratic programming (QP), then we show the solution space of the QP can be equivalently determined by a set of linear inequalities, which can then be efficiently solved by linear programming (LP). Experiments show the robustness and sample efficiency of our methods.
翻译:我们侧重于学习一个由 ReLU (prreReLU-TLRN) 预先激活的双层剩余神经网络: 如果输入 $\ mathbfn{R ⁇ d$x}x} 美元来自支持空间的分布 $\mathb{R ⁇ d$ 和地真真真真真真真假模型是一个预置ReLU-TLRN, 由$\mathb{R__B} ast\left[left (Boldsymbol{B ⁇ st\left}(Poldsymbol{A}Päst\Brasslation{Brex{B}x}xlight}} +\mathbf{xf{x}}}}}} +\mathbxbright}\x}x}x}x}xx}xxxxxxxx}text{}$ $ $ $ 美元, 如果 grot-ruth werubretynetnet we real netnet net net net com real real real deal deal deal lection lection lection legal deal deal deal deal deal deal,, as the a macal deal demax 立, 立好, 一個不以不以不表示 立的立法的立法的立法立法的立法立法的立法立法立法立法立法立法立法立法立法立法立法立法立法, 立法, 立法, 立法, 立法的立法的立法, 立法的內立法的內立法的內立法, 立法, 立法的基的基的立法, 立法的基的內立法的內行法, 立法,它的內立法的立法的立法, 的內行的立法, 的內行的立法的內行的內行的內行法, 的內行法的內行法, 的內行法的內行的內行法的內行法的內行的內行的