Diffusion models that are based on iterative denoising have been recently proposed and leveraged in various generation tasks like image generation. Whereas, as a way inherently built for continuous data, existing diffusion models still have some limitations in modeling discrete data, e.g., languages. For example, the generally used Gaussian noise can not handle the discrete corruption well, and the objectives in continuous spaces fail to be stable for textual data in the diffusion process especially when the dimension is high. To alleviate these issues, we introduce a novel diffusion model for language modeling, Masked-Diffuse LM, with lower training cost and better performances, inspired by linguistic features in languages. Specifically, we design a linguistic-informed forward process which adds corruptions to the text through strategically soft-masking to better noise the textual data. Also, we directly predict the categorical distribution with cross-entropy loss function in every diffusion step to connect the continuous space and discrete space in a more efficient and straightforward way. Through experiments on 5 controlled generation tasks, we demonstrate that our Masked-Diffuse LM can achieve better generation quality than the state-of-the-art diffusion models with better efficiency.
翻译:基于迭代去噪的扩散模型最近已被提出并在各种生成任务中得到应用,如图像生成。然而,作为一种本质上针对连续数据构建的方式,现有的扩散模型在建模离散数据,如语言时仍存在一些限制。例如,通常使用的高斯噪声无法很好地处理离散的破坏,而连续空间中的目标在扩散过程中的稳定性对于文本数据尤其是高维文本数据来说都是有问题的。为了缓解这些问题,我们引入了一种新型的语言建模扩散模型,被称为掩蔽扩散语言模型,它具有更低的训练成本和更好的性能,受到语言特征的启发。具体而言,我们设计了一种语言感知的正向过程,通过有策略的软掩蔽将破坏添加到文本中,以更好地噪声文本数据。此外,我们直接在每个扩散步骤中使用交叉熵损失函数预测分类分布,以更高效、更直接的方式将连续空间和离散空间连接起来。通过对5个受控生成任务的实验,我们证明了我们的掩蔽扩散语言模型可以比最先进的扩散模型实现更好的生成质量,并具有更好的效率。