High-quality HDRIs(High Dynamic Range Images), typically HDR panoramas, are one of the most popular ways to create photorealistic lighting and 360-degree reflections of 3D scenes in graphics. Given the difficulty of capturing HDRIs, a versatile and controllable generative model is highly desired, where layman users can intuitively control the generation process. However, existing state-of-the-art methods still struggle to synthesize high-quality panoramas for complex scenes. In this work, we propose a zero-shot text-driven framework, Text2Light, to generate 4K+ resolution HDRIs without paired training data. Given a free-form text as the description of the scene, we synthesize the corresponding HDRI with two dedicated steps: 1) text-driven panorama generation in low dynamic range(LDR) and low resolution, and 2) super-resolution inverse tone mapping to scale up the LDR panorama both in resolution and dynamic range. Specifically, to achieve zero-shot text-driven panorama generation, we first build dual codebooks as the discrete representation for diverse environmental textures. Then, driven by the pre-trained CLIP model, a text-conditioned global sampler learns to sample holistic semantics from the global codebook according to the input text. Furthermore, a structure-aware local sampler learns to synthesize LDR panoramas patch-by-patch, guided by holistic semantics. To achieve super-resolution inverse tone mapping, we derive a continuous representation of 360-degree imaging from the LDR panorama as a set of structured latent codes anchored to the sphere. This continuous representation enables a versatile module to upscale the resolution and dynamic range simultaneously. Extensive experiments demonstrate the superior capability of Text2Light in generating high-quality HDR panoramas. In addition, we show the feasibility of our work in realistic rendering and immersive VR.
翻译:----
文本与光:零样本文本驱动的高动态范围全景图生成
高质量的高动态范围图像(通常是全景高动态范围图像)是在计算机图形学中创建逼真光照和360度反射的常用方法之一。然而,由于捕捉高动态范围图像的难度,人们想要开发多功能、可控的生成模型来直观地控制生成过程。本文提出了一个零样本文本驱动框架Text2Light来生成4K+分辨率的高动态范围图像,且无需训练数据的对应对齐。给定一个自由文本作为场景的描述,我们用两个专用的步骤来合成相应的高动态范围图像。第一步是使用低动态范围和低分辨率的文本驱动全景图生成,第二步是使用超分辨率的色调映射来将低动态范围全景图的分辨率和动态范围进行放大。具体来说,为了实现零样本文本驱动全景图生成,我们首先构建了双编码本来作为不同环境纹理的离散表示。然后,在预训练的CLIP模型的驱动下,一个文本条件的全局采样器学习从全局编码本中根据输入文本采样出整体语义。此外,一个结构感知的局部采样器学习根据整体语义逐块合成低动态范围全景图。为了实现超分辨率色调映射,我们从低动态范围全景图中导出了一组结构化的潜在代码,作为360度成像的连续表示,并将其锚定到球体上。这种连续表示使得多功能模块能够同时提高分辨率和动态范围。广泛的实验证明了Text2Light在生成高质量高动态范围全景图方面的优越性,同时还展示了我们工作在实际渲染和沉浸式虚拟现实的可行性。