Matrix square roots and their inverses arise frequently in machine learning, e.g., when sampling from high-dimensional Gaussians $\mathcal{N}(\mathbf 0, \mathbf K)$ or whitening a vector $\mathbf b$ against covariance matrix $\mathbf K$. While existing methods typically require $O(N^3)$ computation, we introduce a highly-efficient quadratic-time algorithm for computing $\mathbf K^{1/2} \mathbf b$, $\mathbf K^{-1/2} \mathbf b$, and their derivatives through matrix-vector multiplication (MVMs). Our method combines Krylov subspace methods with a rational approximation and typically achieves $4$ decimal places of accuracy with fewer than $100$ MVMs. Moreover, the backward pass requires little additional computation. We demonstrate our method's applicability on matrices as large as $50,\!000 \times 50,\!000$ - well beyond traditional methods - with little approximation error. Applying this increased scalability to variational Gaussian processes, Bayesian optimization, and Gibbs sampling results in more powerful models with higher accuracy.
翻译:矩阵的根和反面在机器学习中经常出现,例如,当从高维高斯的 $\mathcal{N}(\mathbf 0,\mathbff K) 采样时,当从高维高斯的 $\mathcal{N}(\mathbf 0,\mathbfbf K) 美元中采样时,或用共变量矩阵的倍增法(MVMM)将矢量的 $\mathbffbf b$(b) 或将矢量的 $\mathbfff b$(b) 白化时, 或用共振动量矩阵的 $\ mathbffff k$(b) 美元) 。 虽然现有方法通常需要$O(N(N) 3) $(3) 计算, 但是, 我们引入一种高效的二次算法算算算法, 我们用在大到50, 000 times 50, 000 $. 000(50,\ 000) $. 000) $(远超过传统方法的模型的模型的可变更精确化结果。