Diffusion models in the literature are optimized with various objectives that are special cases of a weighted loss, where the weighting function specifies the weight per noise level. Uniform weighting corresponds to maximizing the ELBO, a principled approximation of maximum likelihood. In current practice diffusion models are optimized with non-uniform weighting due to better results in terms of sample quality. In this work we expose a direct relationship between the weighted loss (with any weighting) and the ELBO objective. We show that the weighted loss can be written as a weighted integral of ELBOs, with one ELBO per noise level. If the weighting function is monotonic, then the weighted loss is a likelihood-based objective: it maximizes the ELBO under simple data augmentation, namely Gaussian noise perturbation. Our main contribution is a deeper theoretical understanding of the diffusion objective, but we also performed some experiments comparing monotonic with non-monotonic weightings, finding that monotonic weighting performs competitively with the best published results.
翻译:文献中的扩散模型使用各种目标进行优化,其中,加权损失函数由加权函数指定,该函数指定每个噪声级别的权重。均匀加权对应于最大化ELBO,这是一种最大似然的基本近似方法。在当前的实践中,由于样本质量更好,扩散模型使用非均匀加权进行优化。在这项工作中,我们揭示了加权损失(采用任何加权方式)和ELBO目标之间的直接关系。我们展示了加权损失可以被写成ELBO的加权积分,其中每个噪声水平对应一个ELBO。如果加权函数是单调的,则加权损失是基于似然的目标:它最大化了简单的数据增强下(即高斯噪声扰动下)的ELBO。我们的主要贡献是对扩散目标的更深入的理论理解,但我们也进行了一些实验,比较单调加权和非单调加权,并发现单调加权的性能与已发表的最佳结果相当。