Current online learning methods suffer issues such as lower convergence rates and limited capability to recover the support of the true features compared to their offline counterparts. In this paper, we present a novel framework for online learning based on running averages and introduce a series of online versions of popular offline methods such as Elastic Net, Minimax Concave Penalty, and Feature Selection with Annealing. The framework can handle an arbitrarily large number of observations with the restriction that the data dimension is not too large, e.g. p<50,000. We prove the equivalence between our online methods and their offline counterparts and give theoretical true feature recovery and convergence guarantees for some of them. In contrast to existing online methods, the proposed methods can extract models with any desired sparsity level at any time. Numerical experiments indicate that our new methods enjoy high true feature recovery accuracy and a fast convergence rate, compared with standard online and offline algorithms. We also show how the running averages framework can be used for model adaptation in the presence of model drift. Finally, we present applications to large datasets where again the proposed framework shows competitive results compared to popular online and offline algorithms.
翻译:当前的在线学习方法存在一些问题,例如,趋同率较低,与离线对应方相比,恢复真实特征支持的能力有限。在本文中,我们提出了一个基于运行平均数的在线学习新框架,并推出一系列受欢迎的离线方法的在线版本,如Elastic Net、Minimax Concave Feem和与Annaaling的功能选择。这个框架可以处理大量武断的观测,但限制数据层面并不太大,例如,p < 50,000。我们证明了我们的在线方法与其离线对应方之间的等同性,并为其中一些方法提供了理论真实特征的恢复和趋同保证。与现有的在线方法不同,拟议方法可以随时提取具有任何预期宽度的模型。数字实验表明,与标准的在线和离线算法相比,我们的新方法具有高度真实的特性恢复准确性和快速趋同率。我们还展示了如何在模型漂移时使用运行平均数框架进行模型调整。最后,我们向大型数据集展示了应用程序,其中拟议的框架再次显示与流行的在线和离线算法的竞争性结果。