Bayesian non-conjugate regression via variational belief updating

We present an efficient semiparametric variational method to approximate the posterior distribution of Bayesian regression models combining subjective prior beliefs with an empirical risk function. Our results apply to all the mixed models predicting the data through a linear combination of the available covariates, including, as special cases, generalized linear mixed models, support vector machines, quantile and expectile regression. The iterative procedure designed for climbing the evidence lower bound only requires closed form updating formulas or the calculation of univariate numerical integrals, when no analytic solutions are available. Neither conjugacy nor elaborate data augmentation strategies are needed. As a generalization, we also extend our methodology in order to account for inducing sparsity and shrinkage priors, with particular attention to the generalizations of the Bayesian Lasso prior. The properties of the derived algorithm are then assessed through an extensive simulation study, in which we compare our proposal with Markov chain Monte Carlo, conjugate mean field variational Bayes and Laplace approximation in terms of posterior approximation accuracy and prediction error. A real data example is then presented through a probabilistic load forecasting application on the US power load consumption data.