LightCity: An Urban Dataset for Outdoor Inverse Rendering and Reconstruction under Multi-illumination Conditions

TL;DR

LightCity introduces a high-fidelity synthetic urban dataset with over 300 HDRI sky maps and 50K images across street-level and aerial views to study multi-illumination effects on intrinsic decomposition, inverse rendering, and urban reconstruction. Built on Blender SceneCity, it provides rich ground-truth attributes (depth, normals, diffuse/roughness, and lighting components) and employs diverse camera sampling and rendering pipelines to simulate realistic indirect lighting and shadows. A comprehensive benchmark shows that current intrinsic decomposition and inverse rendering methods struggle to maintain consistency under varying illumination, while 3D Gaussian Splatting-based approaches offer robust geometry and relighting capabilities, highlighting the dataset’s value for advancing urban scene understanding under complex lighting. LightCity thus enables targeted improvements in robustness to multi-illumination, sim-to-real transfer, and practical urban applications such as autonomous driving and digital twins.

Abstract

Inverse rendering in urban scenes is pivotal for applications like autonomous driving and digital twins. Yet, it faces significant challenges due to complex illumination conditions, including multi-illumination and indirect light and shadow effects. However, the effects of these challenges on intrinsic decomposition and 3D reconstruction have not been explored due to the lack of appropriate datasets. In this paper, we present LightCity, a novel high-quality synthetic urban dataset featuring diverse illumination conditions with realistic indirect light and shadow effects. LightCity encompasses over 300 sky maps with highly controllable illumination, varying scales with street-level and aerial perspectives over 50K images, and rich properties such as depth, normal, material components, light and indirect light, etc. Besides, we leverage LightCity to benchmark three fundamental tasks in the urban environments and conduct a comprehensive analysis of these benchmarks, laying a robust foundation for advancing related research.

Figures (18)

• Figure 1: We present a novel high-quality synthetic urban dataset, named $\mathsf{LightCity}$. Our dataset features complicated urban illumination conditions, including varied illumination, realistic indirect lighting and shadow effects, realistic indirect light and shadow effects, and varying scales with street and aerial image capture.
• Figure 2: Overview. The features of our urban datasets: (a) Diverse variety and flexible control of illuminations. (b) View sampling with varying scales. (c) Multiple properties.
• Figure 3: (a) The HDRI sun height distribution, with 0% representing sea level, covers a wide range of the sun's trajectory throughout the day. (b, d) The HSV distribution of the HDRI maps and albedos spans a wide color space, reflecting diverse lighting and textures. (c) The distribution of rendered shading and RGB variance shows a wide range of brightness and color changes.
• Figure 4: Visualization of decomposed albedo of the same view under multi-illuminations for image intrinsic decomposition.
• Figure 5: Visualization of albedo from inverse rendering.