The matrix completion problem seeks to recover a $d\times d$ ground truth matrix of low rank $r\ll d$ from observations of its individual elements. Real-world matrix completion is often a huge-scale optimization problem, with $d$ so large that even the simplest full-dimension vector operations with $O(d)$ time complexity become prohibitively expensive. Stochastic gradient descent (SGD) is one of the few algorithms capable of solving matrix completion on a huge scale, and can also naturally handle streaming data over an evolving ground truth. Unfortunately, SGD experiences a dramatic slow-down when the underlying ground truth is ill-conditioned; it requires at least $O(\kappa\log(1/\epsilon))$ iterations to get $\epsilon$-close to ground truth matrix with condition number $\kappa$. In this paper, we propose a preconditioned version of SGD that preserves all the favorable practical qualities of SGD for huge-scale online optimization while also making it agnostic to $\kappa$. For a symmetric ground truth and the Root Mean Square Error (RMSE) loss, we prove that the preconditioned SGD converges to $\epsilon$-accuracy in $O(\log(1/\epsilon))$ iterations, with a rapid linear convergence rate as if the ground truth were perfectly conditioned with $\kappa=1$. In our numerical experiments, we observe a similar acceleration for ill-conditioned matrix completion under the 1-bit cross-entropy loss, as well as pairwise losses such as the Bayesian Personalized Ranking (BPR) loss.
翻译:矩阵完成问题试图从对各个元素的观察中回收低等级的美元/ 美元/ 美元/ 美元/ 美元/ 美元/ 美元/ 美元/ 美元/ 美元/ 美元/ 美元/ 美元/ 美元/ 美元/ 美元/ 美元/ 美元/ 美元/ 矩阵完成问题试图从对各个元素的观察中回收低等级的地面真相矩阵。 不幸的是, SGD 在基本地面真相不完善时经历的急剧减慢; 美元/ 美元/ 美元/ 美元/ 美元/ 美元/ 如此之大, 甚至连最简单的全方位矢量矢量矢量操作也变得太昂贵。 本文中, 我们提出了一个SGDD的预设版本, 保存SGD的所有可喜的实用品质, 用于大幅的地面快速优化, 同时将SGD1至 美元/ 个人真相/ 美元/ 美元/ 美元/ 直线性硬度( IMDral_ ral_ ral_ ral_ ral_ ral_ ral_ ral_ ral_ ral_ ral_ ral_ ral_ ral_ ral_ ral_ ral_ ral_ ral_ ral_ ral_ ral_ = = ral_ ral_ = = ral_ ral_ = = = = = = = / = / = = = = = = = = = = = = = = = = = = = = = / = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = 或 或 = = = = = = = = = = = 或 = = =