Text2Light: Zero-Shot Text-Driven HDR Panorama Generation
TOG 2022 (Proc. SIGGRAPH Asia)

Abstract

High-quality HDRIs (High Dynamic Range Images), typically HDR panoramas, are one of the most popular ways to create photorealistic lighting and 360-degree reflections of 3D scenes in graphics. Given the difficulty of capturing HDRIs, a versatile and controllable generative model is highly desired, where laymen users can intuitively control the generation process. However, existing state-of-the-art methods still struggle to synthesize high-quality panoramas for complex scenes. In this work, we propose a zero-shot text-driven framework, Text2Light, to generate 4K+ resolution HDRIs without paired training data. Given a free-form text as the description of the scene, we synthesize the corresponding HDRI with two dedicated steps: 1) text-driven panorama generation in low dynamic range (LDR) and low resolution (LR), and 2) super-resolution inverse tone mapping to scale up the LDR panorama both in resolution and dynamic range. Extensive experiments demonstrate the superior capability of Text2Light in generating high-quality HDR panoramas. In addition, we show the feasibility of our work in realistic rendering and immersive VR.

Framework

We decompose the generation process of HDR panorama into two stages. Stage I translates the input text to LDR panorama based on a dual-codebook discrete representation. First, the input text is mapped to the text embedding by the pre-trained CLIP model. Second, a text-conditioned global sampler learns to sample holistic semantics from the global codebook according to the input text. Then, a structure-aware local sampler synthesizes local patches and composites them accordingly. Stage II upscales the LDR result from Stage I based on structured latent codes as continuous representations. We propose a novel Super-Resolution Inverse Tone Mapping Operator (SR-iTMO) to simultaneously increase the spatial resolution and dynamic range of the panorama.

HDRI Gallery

Hover to view in VR mode. Click the text to open the full viewer and renderer.

closeup photo of concrete stair surrounded by white painted wall	photo of valley
brown wooden hallway during night time	Audience, Auditorium, Conference
body of water surrounded with buildings	seashore during golden hour
photo of orange chairs	brown wooden chairs inside cathedral
ocean waves crashing on shore under blue and white cloudy sky during daytime	white bed linen with white pillow
brown canyon during sunset	gray concrete pathway with wall signages
landscape photography of mountain ranges under purple and pink skies	blue and white hallway with lights turned on in the middle
white ceramic sink beside white ceramic toilet bowl	lined brown pew benches
brown house under stary night	blue and brown wooden counter
road in the middle of grass covered field during day	green grass field with trees and mountains in the distance
green trees on brown rock formation on lake during daytime	concrete road between mountains
road with falling leaves in between of trees	cityscape at night

Video

Citation

Related Links

StyleLight generates HDR indoor panorama from a limited FOV image.
AvatarCLIP proposes a zero-shot text-driven framework for 3D avatar generation and animation.
Text2Human proposes a text-driven controllable human image generation framework.
Relighting4D can relight human actors using the HDRI generated by us.
blender-cli-rendering offers code snippets for rendering 3D assets via command line.