We prove that the evidence lower bound (ELBO) employed by variational auto-encoders (VAEs) admits non-trivial solutions having constant posterior variances under certain mild conditions, removing the need to learn variances in the encoder. The proof follows from an unexpected journey through an array of topics: the closed form optimal decoder for Gaussian VAEs, a proof that the decoder is always smooth, a proof that the ELBO at its stationary points is equal to the exact log evidence, and the posterior variance is merely part of a stochastic estimator of the decoder Hessian. The penalty incurred from using a constant posterior variance is small under mild conditions, and otherwise discourages large variations in the decoder Hessian. From here we derive a simplified formulation of the ELBO as an expectation over a batch, which we call the Batch Information Lower Bound (BILBO). Despite the use of Gaussians, our analysis is broadly applicable -- it extends to any likelihood function that induces a Riemannian metric. Regarding learned likelihoods, we show that the ELBO is optimal in the limit as the likelihood variances approach zero, where it is equivalent to the change of variables formulation employed in normalizing flow networks. Standard optimization procedures are unstable in this limit, so we propose a bounded Gaussian likelihood that is invariant to the scale of the data using a measure of the aggregate information in a batch, which we call Bounded Aggregate Information Sampling (BAGGINS). Combining the two formulations, we construct VAE networks with only half the outputs of ordinary VAEs (no learned variances), yielding improved ELBO scores and scale invariance in experiments. As we perform our analyses irrespective of any particular network architecture, our reformulations may apply to any VAE implementation.
翻译:我们证明,由变异自动编码器(VAE)使用的较低约束度(ELBO)证据(ELBO)表明,在一定的温和条件下,后端差异是非三角解决方案的一部分,在一定的温和条件下,不断出现后端差异,消除了在编码器中学习差异的必要性。证据来自出乎意料的旅程,通过一系列专题:高山VAE的封闭形式最佳解码器,证明解码器总是平滑的,证明ELBO在其固定点与精确的日志证据相等,而后端差异仅仅是在某种随机偏差的测算器中, decoil Coloral Colors Aserian 的测算器中,使用恒定的离差值Ororal AA 的测算器中,在平流中,我们从OBOOO的测算方法中,我们只能进行相应的变现。