PPISP: Physically-Plausible Compensation and Control of Photometric Variations in Radiance Field Reconstruction

TL;DR

This work tackles photometric inconsistencies in radiance-field reconstruction by introducing a differentiable, physically grounded ISP pipeline (PPISP) that disentangles sensor-intrinsic effects from capture-dependent adjustments. A dedicated PPISP controller predicts per-frame ISP parameters for novel views directly from the rendered radiance, enabling realistic cross-view appearance without ground-truth references. The approach yields state-of-the-art performance on standard novel-view benchmarks, can incorporate image metadata when available, and provides interpretable controls for brightness, white balance, and other ISP factors. By enforcing physically plausible constraints and separating modules, PPISP improves photometric consistency while offering practical editing capabilities and robust generalization to unseen viewpoints.

Abstract

Multi-view 3D reconstruction methods remain highly sensitive to photometric inconsistencies arising from camera optical characteristics and variations in image signal processing (ISP). Existing mitigation strategies such as per-frame latent variables or affine color corrections lack physical grounding and generalize poorly to novel views. We propose the Physically-Plausible ISP (PPISP) correction module, which disentangles camera-intrinsic and capture-dependent effects through physically based and interpretable transformations. A dedicated PPISP controller, trained on the input views, predicts ISP parameters for novel viewpoints, analogous to auto exposure and auto white balance in real cameras. This design enables realistic and fair evaluation on novel views without access to ground-truth images. PPISP achieves SoTA performance on standard benchmarks, while providing intuitive control and supporting the integration of metadata when available. The source code is available at: https://github.com/nv-tlabs/ppisp

Figures (10)

• Figure 1: We introduce a differentiable image processing pipeline applied to radiance field reconstruction. By modeling the behavior of conventional cameras, our approach disentangles image formation effects from the rest of the pipeline. Our physically-plausible model admits a controller module that predicts exposure and color changes for novel views.
• Figure 2: Our proposed pipeline applies a sequence of physically-grounded modules to the input reconstructed radiance (exposure offset, chromatic vignetting, linear color correction and non-linear camera response function). Top: all modules except the controller are jointly optimized during the first training phase. Bottom: the controller is then trained to predict per-frame exposure and color correction for novel views while other modules are frozen. The image sequence shows intermediate outputs after each successive module is applied, illustrating the progressive effects of the pipeline.
• Figure 3: Dynamics of the controller module. The predicted exposure offset (inset) depends on the image content of the rendered radiance. Right side: Plot of exposure offsets as predicted for each frame of the caterpillar sequence, with the three displayed frames highlighted.
• Figure 4: Qualitative comparison of novel view synthesis. Row labels indicate datasets and sequences (in italics). Column labels indicate methods. Our method achieves more consistent photometry and better color reproduction across various datasets and sequences. Bottom row: When image metadata such as relative exposure is available, our method can incorporate it to produce a more accurate novel view.
• Figure 5: Qualitative comparison of novel view synthesis, additional examples. Row labels indicate datasets and sequences (in italics). Column labels indicate methods.