Text-to-3D generation has shown rapid progress in recent days with the advent of score distillation, a methodology of using pretrained text-to-2D diffusion models to optimize neural radiance field (NeRF) in the zero-shot setting. However, the lack of 3D awareness in the 2D diffusion models destabilizes score distillation-based methods from reconstructing a plausible 3D scene. To address this issue, we propose 3DFuse, a novel framework that incorporates 3D awareness into pretrained 2D diffusion models, enhancing the robustness and 3D consistency of score distillation-based methods. We realize this by first constructing a coarse 3D structure of a given text prompt and then utilizing projected, view-specific depth map as a condition for the diffusion model. Additionally, we introduce a training strategy that enables the 2D diffusion model learns to handle the errors and sparsity within the coarse 3D structure for robust generation, as well as a method for ensuring semantic consistency throughout all viewpoints of the scene. Our framework surpasses the limitations of prior arts, and has significant implications for 3D consistent generation of 2D diffusion models.
翻译:最近几天,随着分数蒸馏的到来,文本到3D的生成取得了迅速的进展,这是一种使用预先训练的文本到2D的传播模型的方法,在零光环境下优化神经亮度场(NERF)的方法。然而,2D的传播模型缺乏三维意识,使基于分数的蒸馏法无法重建一个合理的三维场景。为了解决这一问题,我们提议3DFuse,这是一个将三维意识纳入预先训练的2D的传播模型的新框架,加强了分数蒸馏法的稳健性和三维一致性。我们通过首先建立一个粗糙的3D结构来实现这一点,先用预测的、特定视图的深度地图作为扩散模型的条件。此外,我们引入了一个培训战略,使二D的传播模型学会处理粗暗的三D结构内的错误和紧张性,以稳健健健的一代,以及确保全场所有观点的语义一致性的方法。我们的框架超越了先前艺术的局限性,并对三维一致生成二D的传播模型产生了重大影响。</s>