The majority of machine learning methods can be regarded as the minimization of an unavailable risk function. To optimize the latter, given samples provided in a streaming fashion, we define a general stochastic Newton algorithm and its weighted average version. In several use cases, both implementations will be shown not to require the inversion of a Hessian estimate at each iteration, but a direct update of the estimate of the inverse Hessian instead will be favored. This generalizes a trick introduced in [2] for the specific case of logistic regression, by directly updating the estimate of the inverse Hessian. Under mild assumptions such as local strong convexity at the optimum, we establish almost sure convergences and rates of convergence of the algorithms, as well as central limit theorems for the constructed parameter estimates. The unified framework considered in this paper covers the case of linear, logistic or softmax regressions to name a few. Numerical experiments on simulated data give the empirical evidence of the pertinence of the proposed methods, which outperform popular competitors particularly in case of bad initializa-tions.
翻译:为了优化后者,我们根据以流态方式提供的样本,定义了一个通用的随机牛顿算法及其加权平均版本。在若干使用案例中,将显示两种应用不要求每次迭代时转换赫森估计值,但直接更新赫森反面估计值将更可取。这概括了[2]中为物流回归这一具体案例引入的伎俩,直接更新了逆赫森的估算值。根据当地强力共和度最佳等温和假设,我们几乎肯定了算法的趋同率和趋同率,以及构建参数估计的中央限制参数值。本文所考虑的统一框架涵盖了线性、后勤性或软形回归等案例。模拟数据的数值实验为拟议方法的相关性提供了经验性证据,这些方法在初始状态不佳的情况下尤其超越了大众竞争者。