Every Camera Effect, Every Time, All at Once: 4D Gaussian Ray Tracing for Physics-based Camera Effect Data Generation

TL;DR

The paper tackles the challenge of generating physically accurate camera-effect data for dynamic scenes, addressing the limitations of pinhole-centric models and costly synthetic pipelines. It introduces 4D Gaussian Ray Tracing (4D-GRT), a two-stage approach that reconstructs dynamic scenes with 4D Gaussian Splatting and renders controllable camera effects via differentiable ray tracing. A synthetic benchmark of 8 indoor scenes across four effects evaluates the method against dynamic NeRF baselines, demonstrating faster rendering and competitive quality. This work enables camera-aware data generation at scale, with practical implications for training robust vision systems under realistic lens distortions and temporal artifacts.

Abstract

Common computer vision systems typically assume ideal pinhole cameras but fail when facing real-world camera effects such as fisheye distortion and rolling shutter, mainly due to the lack of learning from training data with camera effects. Existing data generation approaches suffer from either high costs, sim-to-real gaps or fail to accurately model camera effects. To address this bottleneck, we propose 4D Gaussian Ray Tracing (4D-GRT), a novel two-stage pipeline that combines 4D Gaussian Splatting with physically-based ray tracing for camera effect simulation. Given multi-view videos, 4D-GRT first reconstructs dynamic scenes, then applies ray tracing to generate videos with controllable, physically accurate camera effects. 4D-GRT achieves the fastest rendering speed while performing better or comparable rendering quality compared to existing baselines. Additionally, we construct eight synthetic dynamic scenes in indoor environments across four camera effects as a benchmark to evaluate generated videos with camera effects.

Figures (9)

• Figure 1: We propose 4D Gaussian Ray Tracing (4D-GRT), a novel framework for generating physically-accurate, controllable camera effects in dynamic scenes. (1) Given multi-view video input, our method reconstructs a dynamic scene using 4D Gaussian Splatting (4D-GS) and differentiable ray tracing. (2) We simulate various camera effects with controllable parameters using ray tracing, generating high-quality videos with controllable camera effects.
• Figure 2: We evaluate several state-of-the-art video generation models by specifying camera parameters in prompts to generate videos with specific effects. The results show that these models fail to generate physically accurate videos, instead producing artifacts or incorrect effects. Please refer to the supplementary materials for details.
• Figure 3: The overall pipeline of our model. Given multi-view videos, we optimize the 4D-GS representation through differentiable ray tracing. Then, given camera effect parameters, we can utilize ray tracing to render videos with physically-correct camera effects.
• Figure 4: Qualitative comparison on our synthetic datasets. The artifacts are framed by red rectangles
• Figure 5: Qualitative results of 4D-GRT on the Neural 3D Video dataset li2022neural3dvideosynthesis.