Diffusion models in the literature are optimized with various objectives that are special cases of a weighted loss, where the weighting function specifies the weight per noise level. Uniform weighting corresponds to maximizing the ELBO, a principled approximation of maximum likelihood. In current practice diffusion models are optimized with non-uniform weighting due to better results in terms of sample quality. In this work we expose a direct relationship between the weighted loss (with any weighting) and the ELBO objective. We show that the weighted loss can be written as a weighted integral of ELBOs, with one ELBO per noise level. If the weighting function is monotonic, then the weighted loss is a likelihood-based objective: it maximizes the ELBO under simple data augmentation, namely Gaussian noise perturbation. Our main contribution is a deeper theoretical understanding of the diffusion objective, but we also performed some experiments comparing monotonic with non-monotonic weightings, finding that monotonic weighting performs competitively with the best published results.
翻译:文献中的扩散模型是通过各种不同的目标函数进行优化的,其中加权函数指定每个噪声级别的权重。均匀加权对应于最大似然的有原则的近似ELBO的最大化。目前的做法是通过非均匀加权进行扩散模型优化,从而在样本质量方面获得更好的结果。本文揭示了加权损失(任意加权)和ELBO目标之间的直接关系。我们证明了加权损失可以被写成一组ELBO的加权积分,其中每个噪声级别都对应一个ELBO。如果加权函数是单调的,那么加权损失就是一种基于似然的优化目标:它在简单的数据增强下(即高斯噪声扰动)最大化ELBO。我们的主要贡献是更深入理解扩散目标的理论基础,同时我们还进行了一些实验比较单调和非单调的加权方式,发现单调加权的表现与最佳已发表结果相当。