GS-ProCams: Gaussian Splatting-based Projector-Camera Systems

TL;DR

This work tackles the challenge of view-agnostic projection mapping in projector-camera systems by introducing GS-ProCams, a differentiable framework that represents the scene with 2D Gaussian splats augmented with BRDF attributes. It jointly models projector responses, surface geometry, materials, and residual global illumination through differentiable physically-based rendering, enabling efficient view-agnostic projection mapping and projector compensation without extra light sources. Compared to NeRF-based ProCams, GS-ProCams achieves higher simulation quality while using only 1/10 of the GPU memory for training and about 900x faster inference, and it operates under ambient room lighting. The authors also provide a real-world indoor benchmark and demonstrate superior performance against state-of-the-art view-specific methods, highlighting the approach’s practicality for real-world PM and SAR applications.

Abstract

We present GS-ProCams, the first Gaussian Splatting-based framework for projector-camera systems (ProCams). GS-ProCams is not only view-agnostic but also significantly enhances the efficiency of projection mapping (PM) that requires establishing geometric and radiometric mappings between the projector and the camera. Previous CNN-based ProCams are constrained to a specific viewpoint, limiting their applicability to novel perspectives. In contrast, NeRF-based ProCams support view-agnostic projection mapping, however, they require an additional co-located light source and demand significant computational and memory resources. To address this issue, we propose GS-ProCams that employs 2D Gaussian for scene representations, and enables efficient view-agnostic ProCams applications. In particular, we explicitly model the complex geometric and photometric mappings of ProCams using projector responses, the projection surface's geometry and materials represented by Gaussians, and the global illumination component. Then, we employ differentiable physically-based rendering to jointly estimate them from captured multi-view projections. Compared to state-of-the-art NeRF-based methods, our GS-ProCams eliminates the need for additional devices, achieving superior ProCams simulation quality. It also uses only 1/10 of the GPU memory for training and is 900 times faster in inference speed. Please refer to our project page for the code and dataset: https://realqingyue.github.io/GS-ProCams/.

Figures (13)

• Figure 1: GS-ProCams setup. The intersection point $\mathbf{x}_s$ of the ray from the camera and the projection surface is determined by 2D Gaussians. The direct light from the projector illuminates this point.
• Figure 2: GS-ProCams training pipeline. GS-ProCams uses 2D Gaussians to represent the scene, then obtains the geometry and materials of the projection surface, as well as the global illumination component, through differentiable splatting techniques Huang20242DGS. By using various projection patterns and capturing images from different viewpoints, GS-ProCams jointly optimizes the explicit projector responses and attributes of the 2D Gaussian points listed on the left through physically-based differentiable rendering.
• Figure 3: Visual comparisons of ProCams simulation on the Nepmap synthetic dataset Erel2023Nepmap. The first column displays an object from a novel viewpoint, the second column shows the object under a novel projection pattern, and the third and fourth columns present the results of two methods. We present two of the four scenes here as examples. Compared to Nepmap, our model exhibits finer details and more realistic colors. Moreover, our method outperforms Nepmap in computation and memory efficiency by a significant margin (\ref{['tab: synthetic']}). See supplementary material for more results.
• Figure 4: Learned albedo on the Nepmap synthetic dataset Erel2023Nepmap. We present the albedo maps learned by Nepmap and our GS-ProCams under different novel viewpoints of the same scene, shown in two separate rows. The albedo maps are converted from linear to sRGB color space for a better visual experience. The first column shows the scene under camera co-located light only. The second column depicts the same scene under a uniform white projection. GS-ProCams models the co-located lighting as part of the residual color and includes projector occlusions within the albedo estimation.
• Figure 5: Visual comparisons of ProCams simulation on real-world benchmark dataset. We compare with DeProCams Huang2021DeProcams (view-specific) and DPCS Li2025DPCS (view-specific), for novel projection synthesis. The first row shows a viewpoint included in GS-ProCams training, while the second row shows a novel viewpoint from the same scene that GS-ProCams has never seen during training. The first column shows the same projector input pattern, the second column presents the projection surface under a uniform gray projection and captured in different camera viewpoints, and the remaining columns show the camera-captured ground truth and simulation results. Notably, view-specific methods necessitate repeated data acquisition and training, even for the novel viewpoint, whereas our approach generalizes to novel viewpoints without seeing/retraining on them. Our GS-ProCams consistently produces high-quality simulation results for the novel projection under both trained and novel viewpoints. See supplementary material for more results.