We consider stochastic optimization problems where a smooth (and potentially nonconvex) objective is to be minimized using a stochastic first-order oracle. These type of problems arise in many settings from simulation optimization to deep learning. We present Retrospective Approximation (RA) as a universal sequential sample-average approximation (SAA) paradigm where during each iteration $k$, a sample-path approximation problem is implicitly generated using an adapted sample size $M_k$, and solved (with prior solutions as "warm start") to an adapted error tolerance $\epsilon_k$, using a "deterministic method" such as the line search quasi-Newton method. The principal advantage of RA is that decouples optimization from stochastic approximation, allowing the direct adoption of existing deterministic algorithms without modification, thus mitigating the need to redesign algorithms for the stochastic context. A second advantage is the obvious manner in which RA lends itself to parallelization. We identify conditions on $\{M_k, k \geq 1\}$ and $\{\epsilon_k, k\geq 1\}$ that ensure almost sure convergence and convergence in $L_1$-norm, along with optimal iteration and work complexity rates. We illustrate the performance of RA with line-search quasi-Newton on an ill-conditioned least squares problem, as well as an image classification problem using a deep convolutional neural net.
翻译:我们考虑的是光滑(和潜在非convex)的优化优化问题,在这种情况下,使用随机第一阶或触觉,将平滑(和潜在非convex)目标最小化。这类类型的问题在许多环境中出现,从模拟优化到深层次学习。我们提出回溯性近似(RA),作为通用的相近样本平均近似(SAA)范例,每次迭代(美元)期间,通过调整的样本规模(M美元),隐含地产生样粒子近似问题,并用调整的差错容忍度(先前的解决方案是“热点启动”)解决了(以“温点启动”),用“确定性方法”解决了错误容忍度$($ eepsilon_k),如线搜索准牛顿方法等。RA的主要优点是,从随机近相近(RA)的近似优化,从而可以直接采用现有的确定性算法,从而减轻了为随机环境重新设计算法的必要性。第二个优势是,RA将自身陷入平行的麻烦。我们找出了在 $M_k, kereal rial rial $ riquelum $ 1xillation 和 ration assilgilgilgillation as a orgilgilgilgilvation as axilgilgilvation as a as a lagilgilvolgilgilgilgilgilgild ex.