InFusion: Inject and Attention Fusion for Multi Concept Zero Shot Text based Video Editing

Large text-to-image diffusion models have achieved remarkable success in generating diverse high-quality images in alignment with text prompt used for editing the input image. But, when these models applied to video the main challenge is to ensure temporal consistency and coherence across frames. In this paper, we proposed InFusion, a framework for zero-shot text-based video editing leveraging large pre-trained image diffusion models. Our framework specifically supports editing of multiple concepts with the pixel level control over diverse concepts mentioned in the editing prompt. Specifically, we inject the difference of features obtained with source and edit prompt from U-Net residual blocks in decoder layers, this when combined with injected attention features make it feasible to query the source contents and scale edited concepts along with the injection of unedited parts. The editing is further controlled in fine-grained manner with mask extraction and attention fusion strategy which cuts the edited part from source and paste it into the denoising pipeline for editing prompt. Our framework is a low cost alternative of one-shot tuned models for editing since it does not require training. We demonstrated the complex concept editing with generalised image model (Stable Diffusion v1.5) using LoRA. Adaptation is compatible with all the existing image diffusion techniques. Extensive experimental results demonstrate the effectiveness over existing methods in rendering high-quality and temporally consistent videos.