Audio editing is applicable for various purposes, such as adding background sound effects, replacing a musical instrument, and repairing damaged audio. Recently, some diffusion-based methods achieved zero-shot audio editing by using a diffusion and denoising process conditioned on the text description of the output audio. However, these methods still have some problems: 1) they have not been trained on editing tasks and cannot ensure good editing effects; 2) they can erroneously modify audio segments that do not require editing; 3) they need a complete description of the output audio, which is not always available or necessary in practical scenarios. In this work, we propose AUDIT, an instruction-guided audio editing model based on latent diffusion models. Specifically, AUDIT has three main design features: 1) we construct triplet training data (instruction, input audio, output audio) for different audio editing tasks and train a diffusion model using instruction and input (to be edited) audio as conditions and generating output (edited) audio; 2) it can automatically learn to only modify segments that need to be edited by comparing the difference between the input and output audio; 3) it only needs edit instructions instead of full target audio descriptions as text input. AUDIT achieves state-of-the-art results in both objective and subjective metrics for several audio editing tasks (e.g., adding, dropping, replacement, inpainting, super-resolution). Demo samples are available at https://audit-demo.github.io/.
翻译:音频编辑可用于添加背景音效、更换音乐乐器、修复损坏的音频等各种用途。最近,一些基于扩散的方法通过使用扩散和去噪过程来改善输出音频的文本描述,从而实现了零-shot音频编辑。然而,这些方法仍然存在一些问题:1)它们尚未用于编辑任务并且不能确保良好的编辑效果;2)它们可能错误地修改不需要编辑的音频段;3)它们需要完整的输出音频描述,这并不总是在实际场景中可用或必要的。在本文中,我们提出了AUDIT,一种基于潜在扩散模型的指令引导音频编辑模型。具体来说,AUDIT具有三个主要设计特点:1)我们为不同的音频编辑任务构建三元训练数据(指令、输入音频、输出音频),并使用指令和输入(要编辑)音频作为条件来训练扩散模型,并生成输出(编辑)音频;2)它可以自动学习仅修改需要编辑的片段,方法是比较输入和输出音频之间的差异;3)它仅需要编辑指令而不是完整的目标音频描述作为文本输入。AUDIT在几