3D face reconstruction from a single image is challenging due to its ill-posed nature. Model-based face autoencoders address this issue effectively by fitting a face model to the target image in a weakly supervised manner. However, in unconstrained environments occlusions distort the face reconstruction because the model often erroneously tries to adapt to occluded face regions. Supervised occlusion segmentation is a viable solution to avoid the fitting of occluded face regions, but it requires a large amount of annotated training data. In this work, we enable model-based face autoencoders to segment occluders accurately without requiring any additional supervision during training, and this separates regions where the model will be fitted from those where it will not be fitted. To achieve this, we extend face autoencoders with a segmentation network. The segmentation network decides which regions the model should adapt to by reaching balances in a trade-off between including pixels and adapting the model to them, and excluding pixels so that the model fitting is not negatively affected and reaches higher overall reconstruction accuracy on pixels showing the face. This leads to a synergistic effect, in which the occlusion segmentation guides the training of the face autoencoder to constrain the fitting in the non-occluded regions, while the improved fitting enables the segmentation model to better predict the occluded face regions. Qualitative and quantitative experiments on the CelebA-HQ database and the AR database verify the effectiveness of our model in improving 3D face reconstruction under occlusions and in enabling accurate occlusion segmentation from weak supervision only. Code available at https://github.com/unibas-gravis/Occlusion-Robust-MoFA.
翻译:3D 以单一图像进行面部重建具有挑战性, 原因是它的性质不好。 基于模型的面部自动校正器能够有效地解决这个问题, 使面部模版与目标图像相匹配, 且无需经过严格监督。 然而, 在不受控制的环境中, 面部改造会扭曲面部重建, 因为模型常常被错误地试图适应隐蔽的面部区域。 监督的闭路分解是一个可行的解决方案, 以避免隐蔽的面部区域的安装, 但是它需要大量有注释的培训数据 。 在这项工作中, 我们使基于模型的面部自动校正数能够精确到分解部分的部位, 而无需在培训期间做任何额外的核查。 为了达到这个目的, 我们的面部位将面对自动校正, 分解网络决定了模型应该适应哪些区域, 在包括像素模型和调整模型的面部之间实现平衡, 排除了像素, 以便模型的装配制不会对面部的面部位进行负面的影响, 并且更精确地对Qelels- 进行整个重建监督 。 。 在升级的部部分中, 改进了节面部校正的校正的校正的校正的校正的校正, 将产生一个协同效果, 。