We consider a general class of regression models with normally distributed covariates, and the associated nonconvex problem of fitting these models from data. We develop a general recipe for analyzing the convergence of iterative algorithms for this task from a random initialization. In particular, provided each iteration can be written as the solution to a convex optimization problem satisfying some natural conditions, we leverage Gaussian comparison theorems to derive a deterministic sequence that provides sharp upper and lower bounds on the error of the algorithm with sample-splitting. Crucially, this deterministic sequence accurately captures both the convergence rate of the algorithm and the eventual error floor in the finite-sample regime, and is distinct from the commonly used "population" sequence that results from taking the infinite-sample limit. We apply our general framework to derive several concrete consequences for parameter estimation in popular statistical models including phase retrieval and mixtures of regressions. Provided the sample size scales near-linearly in the dimension, we show sharp global convergence rates for both higher-order algorithms based on alternating updates and first-order algorithms based on subgradient descent. These corollaries, in turn, yield multiple consequences, including: (a) Proof that higher-order algorithms can converge significantly faster than their first-order counterparts (and sometimes super-linearly), even if the two share the same population update and (b) Intricacies in super-linear convergence behavior for higher-order algorithms, which can be nonstandard (e.g., with exponent 3/2) and sensitive to the noise level in the problem. We complement these results with extensive numerical experiments, which show excellent agreement with our theoretical predictions.
翻译:我们考虑的是一般分布的共变数回归模型的一般类别,以及相关的非混凝土问题。我们从数据中匹配这些模型。我们开发了一种从随机初始化中分析这一任务迭代算法趋同的通用配方。特别是,如果每个迭代可以写成,作为满足某些自然条件的共变优化问题的解决方案,我们利用高斯比较理论来得出一个确定序列,该序列在样本分离的算法错误上下提供了锐利的分解范围。关键是,这一确定性序列准确地捕捉了算法的趋同率和定值定值中最终差值下层的趋同率。我们开发了一种通用的“人口”序列,其结果来自无限抽样优化限。我们运用了我们的总框架来为流行的统计模型的参数估算得出若干具体后果,包括阶段检索和回归的组合。如果在范围上,我们展示了基于交替更新和基于定序系统下缩缩略图的首级算法的更高级算法和一级算法的一致率率率率,这些数值序列中,其次级算算算算值可以明显地显示为: 递增后算结果,其次序列中的结果可以显示为: 递增后演算结果, 级序列中,甚至为: 递增后算结果, 递增后演算结果, 递增后演算算算法可以显示为递归为递归为: 递归为: 级为: 级为:, 级, 级, 递归为: 递归为: 递增后算结果, 递增后算算算结果为:,, 级为 级为: 递算算为 递算为: 算为: 算为 算为: 算为 算为 算算算为 算为 级为 算为 级为 级为 级为 级为 级为 算为 算为 级为 级为 算为, 级, 级为 级为 级为 算算算算算为 级为 级为 级为 级为 级为 级为 级为 算算算值为 级为 级为 级为 级为 级为 级为