Iterative gradient-based algorithms have been increasingly applied for the training of a broad variety of machine learning models including large neural-nets. In particular, momentum-based methods, with accelerated learning guarantees, have received a lot of attention due to their provable guarantees of fast learning in certain classes of problems and multiple algorithms have been derived. However, properties for these methods hold only for constant regressors. When time-varying regressors occur, which is commonplace in dynamic systems, many of these momentum-based methods cannot guarantee stability. Recently, a new High-order Tuner (HT) was developed for linear regression problems and shown to have 1) stability and asymptotic convergence for time-varying regressors and 2) non-asymptotic accelerated learning guarantees for constant regressors. In this paper, we extend and discuss the results of this same HT for general convex loss functions. Through the exploitation of convexity and smoothness definitions, we establish similar stability and asymptotic convergence guarantees. Finally, we provide numerical simulations supporting the satisfactory behavior of the HT algorithm as well as an accelerated learning property.
翻译:在培训包括大型神经网在内的各种机器学习模型时,越来越多地采用基于迭代梯度的算法;特别是,动力基方法,加上加速学习的保证,由于在某些类别的问题和多种算法中可被证实的快速学习的保证,因此受到了很多关注;然而,这些方法的特性只对不断递减者具有特性;在动态系统中常见的时间变化递减者出现时,许多这些基于动力的方法无法保证稳定性;最近,针对线性回归问题开发了新的高阶图纳(HT),显示有:(1) 时间变化递减递减者的稳定性和无刺激性趋同;(2) 持续递减者的不被动加速学习保证;在本文件中,我们扩展和讨论相同的HT结果,用于一般 convex损失功能;通过利用调和平稳定义,我们建立了类似的稳定性和平衡性保证。最后,我们提供了数字模拟,支持HT算法的令人满意的行为,作为加速学习财产。