DPCS: Path Tracing-Based Differentiable Projector-Camera Systems

TL;DR

DPCS introduces a path-tracing-based differentiable projector-camera system to simulate ProCams with explicit, interpretable scene parameters. By formulating forward rendering as $I_c = \mathcal{R}(I_p, \boldsymbol{\theta})$ and enabling inverse rendering $\{I_p, \boldsymbol{\theta}\} = \mathcal{R}^{-1}(I_c)$, it jointly estimates BRDF, radiometric responses, and geometry from a small set of paired projections and captures. The approach yields improved relighting, projector compensation, BRDF estimation, and novel-scene simulation compared to neural baselines, while preserving physical interpretability and requiring fewer training samples; it also supports editing scene parameters for SAR applications. Limitations include single-view optimization and computational demands, with future work aiming at multi-view guidance, improved BRDF decomposition, and potential real-time acceleration through optimized denoising and rendering. Overall, DPCS demonstrates that differentiable, physically-based rendering can robustly model complex ProCams interactions, enabling accurate, editable simulations for SAR tasks.

Abstract

Projector-camera systems (ProCams) simulation aims to model the physical project-and-capture process and associated scene parameters of a ProCams, and is crucial for spatial augmented reality (SAR) applications such as ProCams relighting and projector compensation. Recent advances use an end-to-end neural network to learn the project-and-capture process. However, these neural network-based methods often implicitly encapsulate scene parameters, such as surface material, gamma, and white balance in the network parameters, and are less interpretable and hard for novel scene simulation. Moreover, neural networks usually learn the indirect illumination implicitly in an image-to-image translation way which leads to poor performance in simulating complex projection effects such as soft-shadow and interreflection. In this paper, we introduce a novel path tracing-based differentiable projector-camera systems (DPCS), offering a differentiable ProCams simulation method that explicitly integrates multi-bounce path tracing. Our DPCS models the physical project-and-capture process using differentiable physically-based rendering (PBR), enabling the scene parameters to be explicitly decoupled and learned using much fewer samples. Moreover, our physically-based method not only enables high-quality downstream ProCams tasks, such as ProCams relighting and projector compensation, but also allows novel scene simulation using the learned scene parameters. In experiments, DPCS demonstrates clear advantages over previous approaches in ProCams simulation, offering better interpretability, more efficient handling of complex interreflection and shadow, and requiring fewer training samples.

Figures (16)

• Figure 1: Our physically-based differentiable simulation framework. First, the scene is acquired using structured light (SL) huang2020fast to calibrate and reconstruct the surface as a point cloud, which is then utilized to reconstruct the surface into a mesh format. Then, a forward differentiable rendering works to simulate the light transport of the ProCams using predefined scene parameters which contain the surface reflectance, projector response function, and camera response function. Other physical factors like the white balance coefficients, can also be defined for more accurate simulation. The forward rendering approach gives noisy rendered images of the different projection lighting captured by the camera which can be used to calculate pixel loss to the real capturing. Once a denoising filter is applied to the noisy rendered image, it can be leveraged in a gradient-based optimization to minimize the pixel loss between the denoised rendered images and camera-captured images by differentiating the virtual ProCams physical parameters.
• Figure 2: ProCams imaging process. We assume that the scene contains only a projected light source. The scene is illuminated by the direct lighting $\mathbf{L}_\text{p}$ emitted by the projected light source and the indirect lighting $\mathbf{L}_\text{i}$ resulting from multiple reflections, which is ultimately captured by the camera. The nonlinear transformations of the camera and projector are expressed using gamma functions.
• Figure 3: Projector compensation pipeline. Once our DPCS is trained, we can get the compensated projector input image by differentiating the projector input image such that the rendering result is close to the desired appearance.
• Figure 4: Qualitative comparison on ProCams relighting. We present three scenes under different novel projector input patterns. Each image is provided with two zoomed-in patches for detailed comparison. The 1st column represents the projector input, the 2nd shows the camera-captured scenes, the 3rd to 5th present the relighting results of different methods, and the last column is the camera-captured ground truth, i.e., the projection of the 1st onto the 2nd.
• Figure 5: Qualitative comparison of real projector compensation. Columns $1$ to $3$ display the projection surface, the desired image as perceived by viewers, and the uncompensated projection, respectively. Subsequent columns present real camera-captured outcomes from various compensation techniques. DPCS (simulated) represents the simulated compensated result in the renderer.