Modern methods for Bayesian regression beyond the Gaussian response setting are computationally impractical or inaccurate in high dimensions. As discussed in recent literature, bypassing this trade-off is still an open problem even in basic binary regression models, and there is limited theory on the quality of variational approximations in high-dimensional settings. To address this gap, we study the approximation accuracy of routine-use mean-field variational Bayes in high-dimensional probit regression with Gaussian priors, obtaining new and practically relevant results on the pathological behavior of this strategy in uncertainty quantification, estimation and prediction, that also suggest caution against maximum a posteriori estimates when p>n. Motivated by these results, we develop a new partially-factorized variational approximation for the posterior distribution of the probit coefficients that leverages a representation with global and local variables but, unlike for classical mean-field assumptions, it avoids a fully factorized approximation, and instead assumes a factorization only for local variables. We prove that the resulting approximation belongs to a tractable class of unified skew-normal densities that incorporates skewness and, unlike for state-of-the-art mean-field solutions, converges to the exact posterior density as p goes to infinity. To solve the variational optimization problem, we derive a tractable CAVI algorithm that easily scales to p in tens of thousands, and provably requires a number of iterations converging to 1 as p goes to infinity. Such findings are also illustrated in extensive empirical studies where our new solution is shown to improve the accuracy of mean-field variational Bayes for any n and p, with the magnitude of these gains being remarkable in those high-dimensional p>n settings where state-of-the-art methods are computationally impractical.
翻译:高山响应设置之外巴伊西亚回归的现代方法在高山响应设置中是计算不切实际或不准确的。正如最近文献中所讨论的那样,绕过这一权衡,即使在基本的二进制回归模型中,仍然是一个尚未解决的问题,而且对于高维环境中的变差近似值质量的理论也很有限。为了解决这一差距,我们研究高空前科中常规使用平均场变差贝亚的近似准确性,在高高原前科中获取新的和实际相关的结果,在不确定性的量化、估计和预测中,这一战略的病理行为中,获得新的和现实相关的结果,这也表明,要谨慎避免在Postiorial 估算时出现最大程度的顺差值估计。受这些结果的激励,我们开发了一个新的局部离差差差差差差差差值近值的理论,在高端平流的轨道中,我们产生的近似易变差值直线的直线度直线直径直径直线直径直径直径直径直径直达。