ISETHDR: A Physics-based Synthetic Radiance Dataset for High Dynamic Range Driving Scenes

TL;DR

The paper tackles the challenge of capturing high dynamic range driving scenes under HDR conditions by introducing a physics-based end-to-end simulator that models scene spectral radiance, optics, and sensor responses. It advances a labeled spectral HDR driving dataset comprised of 2000 light-group scenes and provides open-source software to simulate end-to-end imaging from radiance to sensor data. Through a comparative study of a split-pixel 3-capture sensor and an RGBW sensor, the work demonstrates how physics-based rendering and end-to-end modeling can reveal regime-specific advantages (e.g., flare mitigation with split-pixel, improved low-light quality with RGBW) for automotive imaging. The dataset and digital twin framework enable quantitative evaluation and design optimization of HDR imaging systems for safer, more robust driving under challenging lighting conditions.

Abstract

This paper describes a physics-based end-to-end software simulation for image systems. We use the software to explore sensors designed to enhance performance in high dynamic range (HDR) environments, such as driving through daytime tunnels and under nighttime conditions. We synthesize physically realistic HDR spectral radiance images and use them as the input to digital twins that model the optics and sensors of different systems. This paper makes three main contributions: (a) We create a labeled (instance segmentation and depth), synthetic radiance dataset of HDR driving scenes. (b) We describe the development and validation of the end-to-end simulation framework. (c) We present a comparative analysis of two single-shot sensors designed for HDR. We open-source both the dataset and the software.

Figures (9)

• Figure 1: Overview. A physics-based end-to-end image systems simulator was developed to synthesize spectral radiance of complex scenes and to model both image formation and image sensor designs. This work introduces a driving scene dataset (top row) comprising renderings illuminated by distinct light groups (sky, headlights, streetlights, and other sources). Daytime and nighttime scene spectral radiance were simulated by linearly combining these light group renderings (bottom left and middle images, respectively). Pixel-level object labels and depth maps are provided for each scene (right).
• Figure 2: Beam angle control. To accurately simulate various light sources such as headlights, taillights, and streetlights, we extended the PBRT arealight model to include controllable beam angle parameters (top). The bottom image demonstrates the limited beam angle area lights for car headlights.
• Figure 3: Scene light groups. The dataset comprises 2000 scenes, each defined by four spectral radiance maps representing illumination by the sky, headlights, streetlights, and other light sources (e.g., tail lights, bicycle lights). To simulate various lighting conditions, the four maps are combined with different weights. For example, a daytime scene (left) has a bright sky and headlights, while a nighttime scene (right) has a darker sky with prominent headlights and streetlights. Using a lens model incorporating aperture and scratch effects (but excluding inter-reflections), scene radiance is converted to sensor irradiance. The graph on the right illustrates the illumination profile across a horizontal line. Notably, headlight intensity remains constant between day and night, while reduced skylight lowers image contrast in darker areas. The software includes tools to select the appropriate weights for achieving the desired dynamic range and low-light conditions.(lightGroupDynamicRangeSet.m).
• Figure 4: Flare model. The figure depicts a series of simulated scenes featuring an array of bright lights, resembling headlights, with a dark image in the background. The bright light intensities each step down by a factor of 10 across the image. Each scene was rendered using distinct flare parameters. The number of aperture blades increases from four (leftmost column) to a circular aperture (rightmost column). The density of simulated dust and scratches varies from high (top row) to minimal (bottom row).
• Figure 5: Simulated and measured nighttime driving scenes.Left: Images in this column were captured in rapid succession by a Google Pixel 4a, with exposure duration increasing from top to bottom (see inset). Right: Images in this column were simulated, using a model of the Google Pixel 4a Lyu2022-validation. The spatial extent of the flare, and the corresponding extent of the sensor saturation, are very similar when comparing the two columns. The red boxes outline two vulnerable road users: a cyclist (left) and motorcyclist (right). As exposure duration increases from 5 to 40 ms both cyclists become more visible. At the longest duration, the flare - arising from headlights behind the cyclists - expands and masks a significant part of these vulnerable road users.