Denoising diffusion probabilistic models (DDPMs) have shown impressive results on sequence generation by iteratively corrupting each example and then learning to map corrupted versions back to the original. However, previous work has largely focused on in-place corruption, adding noise to each pixel or token individually while keeping their locations the same. In this work, we consider a broader class of corruption processes and denoising models over sequence data that can insert and delete elements, while still being efficient to train and sample from. We demonstrate that these models outperform standard in-place models on an arithmetic sequence task, and that when trained on the text8 dataset they can be used to fix spelling errors without any fine-tuning.
翻译:否认扩散概率模型(DDPMs)通过反复腐蚀每个实例,然后学习将腐败版本映射回原始版本,在序列生成方面显示了令人印象深刻的结果。 但是,先前的工作主要侧重于现场腐败,在保持其位置的同时,给每个像素或符号都增加噪音,同时保持其位置不变。 在这项工作中,我们考虑的是更广泛的腐败过程类别,以及相对于可以插入和删除元素的序列数据进行分解的模式,同时仍然能有效地进行培训和取样。 我们证明这些模型在算术序列任务上优于当地标准模型,在接受文本8数据集培训时,它们可以用来在不作任何微调的情况下修正拼写错误。