Sequential Attention GAN for Interactive Image Editing via Dialogue

We introduce a new task - Interactive Image Editing via conversational language, where users can guide an agent to edit images via multi-turn dialogue. In each dialogue turn, the agent takes a source image and a natural language description as the input, and generates a modified image following the textual description. Two new datasets are introduced for this task (Zap-Seq and DeepFashion-Seq), which contain multi-turn dialog sessions with crowdsourced image-description sequences. The main challenges in this sequential and interactive image generation task are two-fold: 1) contextual consistency between a generated image and the given textual description; 2) step-by-step region-level modification to maintain visual consistency across the image sequence. To address these challenges, we propose a novel Sequential Attention Generative Adversarial Network (SeqAttnGAN) framework, which applies a neural state tracker to encode the previous image and the textual description in each dialogue turn, and uses a GAN framework to generate a modified version of the image that is consistent with the dialogue context and preceding images. To achieve better region-specific refinement, we also introduce a sequential attention mechanism into the model. Experiments on Zap-Seq and DeepFashion-Seq datasets show that the proposed SeqAttnGAN model outperforms state-of-the-art approaches on the interactive image editing task across all evaluation metrics on visual quality, image sequence coherence and text-image consistency.