We propose the first joint audio-video generation framework that brings engaging watching and listening experiences simultaneously, towards high-quality realistic videos. To generate joint audio-video pairs, we propose a novel Multi-Modal Diffusion model (i.e., MM-Diffusion), with two-coupled denoising autoencoders. In contrast to existing single-modal diffusion models, MM-Diffusion consists of a sequential multi-modal U-Net for a joint denoising process by design. Two subnets for audio and video learn to gradually generate aligned audio-video pairs from Gaussian noises. To ensure semantic consistency across modalities, we propose a novel random-shift based attention block bridging over the two subnets, which enables efficient cross-modal alignment, and thus reinforces the audio-video fidelity for each other. Extensive experiments show superior results in unconditional audio-video generation, and zero-shot conditional tasks (e.g., video-to-audio). In particular, we achieve the best FVD and FAD on Landscape and AIST++ dancing datasets. Turing tests of 10k votes further demonstrate dominant preferences for our model. The code and pre-trained models can be downloaded at https://github.com/researchmm/MM-Diffusion.
翻译:我们提出了第一个联合音视频生成框架,可以同时提供引人入胜的观听体验,以实现高质量逼真的视频。为了生成联合音视频对,我们提出了一种新的多模态扩散模型(即MM-Diffusion),其中包含两个耦合的去噪自编码器。与现有的单模态扩散模型不同,MM-Diffusion由一个序列多模态U-Net组成,通过设计进行联合去噪过程。音频和视频的两个子网络逐步从高斯噪声学习生成对其的音视频对。为了确保跨模式的语义一致性,我们提出了一种新型基于随机移位的注意力块,将两个子网络连接起来,实现高效的跨模态对齐,从而增强彼此的音视频保真度。广泛的实验显示,在无条件音视频生成和零样本条件任务(例如,视频到音频)方面均取得了卓越的结果。特别是在风景和AIST ++跳舞数据集上实现了最佳的FVD和FAD。10000次图灵测试进一步证明了我们模型的优势。代码和预训练模型可在https://github.com/researchmm/MM-Diffusion下载。