We introduce an end-to-end learning framework for image-to-image composition, aiming to plausibly compose an object represented as a cropped patch from an object image into a background scene image. As our approach emphasizes more on semantic and structural coherence of the composed images, rather than their pixel-level RGB accuracies, we tailor the input and output of our network with structure-aware features and design our network losses accordingly, with ground truth established in a self-supervised setting through the object cropping. Specifically, our network takes the semantic layout features from the input scene image, features encoded from the edges and silhouette in the input object patch, as well as a latent code as inputs, and generates a 2D spatial affine transform defining the translation and scaling of the object patch. The learned parameters are further fed into a differentiable spatial transformer network to transform the object patch into the target image, where our model is trained adversarially using an affine transform discriminator and a layout discriminator. We evaluate our network, coined SAC-GAN, for various image composition scenarios in terms of quality, composability, and generalizability of the composite images. Comparisons are made to state-of-the-art alternatives, including Instance Insertion, ST-GAN, CompGAN and PlaceNet, confirming superiority of our method.
翻译:我们为图像到图像的构成引入了端到端学习框架, 目的是将一个标的物体从对象图像中成成成块的补丁, 从对象图像到背景场景图像。 由于我们的方法更多地强调组成图像的语义和结构一致性, 而不是像素级的 RGB 光谱, 我们用结构认知特性来调整网络的输入和输出, 并据此设计网络损失, 以自监督的方式通过对象裁剪设置来建立地面真相。 具体地说, 我们的网络从输入场景图像、 从输入对象补丁的边缘和硅状标码中采集的语义布局特征, 以及作为输入的隐含代码, 并生成一个 2D 空间线来改变对象补丁的翻译和缩放。 我们所学的参数被进一步注入一个不同的空间变异变异网络, 以便把对象补丁制成目标图像, 我们的模型通过对调方式, 转换导导师和布局分析器。 我们评估我们的网络, 硬化的SAC- GAN- GAN 的边框度, 以及各种图像的可比较性, 的可比较性, 的可比较性, 的模型的可变性, 的可变性, 的可变性, 的可变性, 的可变性, 的可变性, 的可变性, 和性, 和性, 的可变性, 的可变性, 的可变性图像比性, 的可变性, 和性, 的可变式的可变性, 的可变性, 的可变性, 。