We investigate the connections between sparse approximation methods for making kernel methods and Gaussian processes (GPs) scalable to massive data, focusing on the Nystr\"om method and the Sparse Variational Gaussian Processes (SVGP). While sparse approximation methods for GPs and kernel methods share some algebraic similarities, the literature lacks a deep understanding of how and why they are related. This is a possible obstacle for the communications between the GP and kernel communities, making it difficult to transfer results from one side to the other. Our motivation is to remove this possible obstacle, by clarifying the connections between the sparse approximations for GPs and kernel methods. In this work, we study the two popular approaches, the Nystr\"om and SVGP approximations, in the context of a regression problem, and establish various connections and equivalences between them. In particular, we provide an RKHS interpretation of the SVGP approximation, and show that the Evidence Lower Bound of the SVGP contains the objective function of the Nystr\"om approximation, revealing the origin of the algebraic equivalence between the two approaches. We also study recently established convergence results for the SVGP and how they are related to the approximation quality of the Nystr\"om method.
翻译:我们调查了制造内核方法的稀薄近似方法与高森进程(GPs)之间联系,这些方法可以伸缩到大量数据中,重点是Nystr\'om 方法和Sprass Varizational Gaussian 进程(SVGP) 。 虽然GPs和内核方法的稀薄近近似方法有某些代数相似之处,但文献却缺乏对两者之间联系的方式和原因的深刻理解。这是GP和内核社区之间沟通的可能障碍,使得很难将结果从一方转移到另一方。 我们的动机是通过澄清GPs和内核方法的稀散近似方法之间的联系来消除这一可能的障碍。 在这项工作中,我们研究了两种流行的方法,即Nystr\'om 和SVGPs 近似方法,在回归问题的背景下,并确定了它们之间的各种联系和等值。特别是,我们提供了对SVGPs近似方法的 RKHs解释, 并表明SVGPs Bown Buround 包含Nystrom近似方法的客观功能, 揭示了Nystrat\\\ milling 和SGVs 相关结果的来源。