In this paper, we study and prove the non-asymptotic superlinear convergence rate of the Broyden class of quasi-Newton methods including Davidon--Fletcher--Powell (DFP) method and Broyden--Fletcher--Goldfarb--Shanno (BFGS) method. The asymptotic superlinear convergence rate of these quasi-Newton methods has been extensively studied, but their explicit finite time local convergence rate is not fully investigated. In this paper, we provide a finite time (non-asymptotic) convergence analysis for BFGS and DFP methods under the assumptions that the objective function is strongly convex, its gradient is Lipschitz continuous, and its Hessian is Lipschitz continuous only in the direction of the optimal solution. We show that in a local neighborhood of the optimal solution, the iterates generated by both DFP and BFGS converge to the optimal solution at a superlinear rate of $(1/k)^{k/2}$, where $k$ is the number of iterations. We also prove the same local superlinear convergence rate in the case that the objective function is self-concordant. Numerical experiments on different objective functions confirm our explicit convergence rates. Our theoretical guarantee is one of the first results that provide a non-asymptotic superlinear convergence rate for DFP and BFGS quasi-Newton methods.
翻译:在本文中,我们研究并证明准牛顿方法(包括Davidon-Fletcher-Powell(DFP)方法和Broyden-Fletcher-Goldfarb-Shanno(BFGS)方法)的非表面超线性超线性趋同率。这些准牛顿方法(BFGS)的非表面性超线性超线性趋同率得到了广泛的研究,但并未充分调查这些方法的明确有限时间当地趋同率。在本文中,我们为BFGS和DFP方法提供了有限的时间(非表面性)趋同率分析,其假设是:目标功能是很强的 convex,其梯度是Lipschitz-Fletcher-Goldfarb-Shanno(Goldforforb-Shanno)方法,以及BSHiscitzt(Lipschitz)方法只是朝着最佳解决办法的方向持续走下去。我们显示,在最佳解决办法的当地附近地区,DFP和BFGS产生的超线性超线性超线性超线性超线性趋同率率($k) ) 和最优化解决办法最接近最佳解决办法与最佳解决办法相趋同于最接近的融合率。在美元($1/Link-col-col-col-col-col-colental-col-col-colental) 的精确率中,其中, 和我们的不相趋同为一等的理论性试验率。