Recent deep generative inpainting methods use attention layers to allow the generator to explicitly borrow feature patches from the known region to complete a missing region. Due to the lack of supervision signals for the correspondence between missing regions and known regions, it may fail to find proper reference features, which often leads to artifacts in the results. Also, it computes pair-wise similarity across the entire feature map during inference bringing a significant computational overhead. To address this issue, we propose to teach such patch-borrowing behavior to an attention-free generator by joint training of an auxiliary contextual reconstruction task, which encourages the generated output to be plausible even when reconstructed by surrounding regions. The auxiliary branch can be seen as a learnable loss function, i.e. named as contextual reconstruction (CR) loss, where query-reference feature similarity and reference-based reconstructor are jointly optimized with the inpainting generator. The auxiliary branch (i.e. CR loss) is required only during training, and only the inpainting generator is required during the inference. Experimental results demonstrate that the proposed inpainting model compares favourably against the state-of-the-art in terms of quantitative and visual performance.
翻译:最近的深层基因描述方法使用关注层,使发电机能够明确借用已知区域的特征补丁以完成一个缺失的区域。由于缺乏对缺失区域和已知区域之间通信的监管信号,它可能无法找到适当的参考特征,这往往导致结果中的文物。此外,它计算出在推论过程中整个特征图的双向相似之处,从而产生一个巨大的计算间接费用。为了解决这一问题,我们提议通过联合培训辅助背景重建任务,将这种补丁借款行为教给一个无注意力生成器,这鼓励了即使由周围区域重建后产生的产出也是可信的。辅助分支可被视为一种可学习的损失功能,即被称为背景重建(CR)损失,其查询参考特征相似性和参考重建器与涂料生成器共同优化。辅助分支(即CR损失)仅在培训期间需要,而在推论期间只需要涂料生成器。实验结果表明,拟议的图样模型在定量和性能方面优于状态。