We provide (high probability) bounds on the condition number of random feature matrices. In particular, we show that if the complexity ratio $\frac{N}{m}$ where $N$ is the number of neurons and $m$ is the number of data samples scales like $\log^{-3}(N)$ or $\log^{3}(m)$, then the random feature matrix is well-conditioned. This result holds without the need of regularization and relies on establishing a bound on the restricted isometry constant of the random feature matrix. In addition, we prove that the risk associated with regression problems using a random feature matrix exhibits the double descent phenomenon and that this is an effect of the double descent behavior of the condition number. The risk bounds include the underparameterized setting using the least squares problem and the overparameterized setting where using either the minimum norm interpolation problem or a sparse regression problem. For the least squares or sparse regression cases, we show that the risk decreases as $m$ and $N$ increase, even in the presence of bounded or random noise. The risk bound matches the optimal scaling in the literature and the constants in our results are explicit and independent of the dimension of the data.
翻译：我们提供了随机特征矩阵条件数的(高概率)界限。 特别是, 我们显示, 如果复杂比率 $\ frac{ n ⁇ m} $( 美元) 代表神经元的数量和 美元美元 美元 代表数据样本数量, 比如 $\ log ⁇ - 3} (N) 美元 或 $\ log} (m) 美元, 那么随机特征矩阵是良好的条件。 因此, 随机特征矩阵不需要正规化, 并依赖于对随机特征矩阵的限制性偏差常值设定一个约束值。 此外, 我们证明, 使用随机特征矩阵的回归问题带来的风险显示出双向下降现象, 这是条件号双向下降行为的影响。 风险界限包括使用最小正方位问题或 $\ $( $) 或 $( $) $( $) $) 和 $( $) $( $) 和 $( $( $) $( $) $( $) $( $) $ ( $) $( $) $( $) $ ( $) $ ( $) $) ( $) 。 。 和 这是条件数 的双向下移 和 的双向的双向现象的效应现象的影响。 。 。 风险绑定在文献中, 和 和 和 和 常数中, 等数据 独立数据 。
相关内容
微信扫码咨询专知VIP会员