Gaussian Splatting on the Move: Blur and Rolling Shutter Compensation for Natural Camera Motion

TL;DR

This work targets high-quality 3D scene reconstruction from handheld video by explicitly modeling motion blur and rolling shutter within Gaussian Splatting. It introduces a differentiable rendering pipeline that integrates camera trajectory, using velocities from visual–inertial odometry (VIO) and optimizing poses during reconstruction, with a screen-space approximation to efficiently handle exposure-time effects. The approach avoids adding learned blur modules, instead relying on per-frame motion integration and pixel-velocity rasterization, yielding quantitative gains (PSNR, SSIM, LPIPS) over 3DGS baselines and BAD-NeRF on synthetic data and robust performance on real smartphone data. The method demonstrates practical improvements for naturalistic capture scenarios, offers ablations and timing analyses, and provides public code and datasets for reproducibility and further research in differentiable rendering under realistic imaging imperfections.

Abstract

High-quality scene reconstruction and novel view synthesis based on Gaussian Splatting (3DGS) typically require steady, high-quality photographs, often impractical to capture with handheld cameras. We present a method that adapts to camera motion and allows high-quality scene reconstruction with handheld video data suffering from motion blur and rolling shutter distortion. Our approach is based on detailed modelling of the physical image formation process and utilizes velocities estimated using visual-inertial odometry (VIO). Camera poses are considered non-static during the exposure time of a single image frame and camera poses are further optimized in the reconstruction process. We formulate a differentiable rendering pipeline that leverages screen space approximation to efficiently incorporate rolling-shutter and motion blur effects into the 3DGS framework. Our results with both synthetic and real data demonstrate superior performance in mitigating camera motion over existing methods, thereby advancing 3DGS in naturalistic settings.

Figures (9)

• Figure 1: 3DGS reconstructions from the synthetic cozyroom scene under different simulated effects with (bottom row) and without (top row) our compensation. The corresponding numerical results are given in \ref{['tab:synthetic']}. The bottom right image is visually indistinguishable from the reference (PSNR 36.2).
• Figure 2: Further 3DGS reconstructions from the synthetic factory scene. The Splatfacto method acting as a baseline. The corresponding numerical results are given in \ref{['tab:synthetic']}. In this case, the pose optimization case converged to a local minimum.
• Figure 3: Screen space approximation incorporates the motion during the frame exposure interval into our model by capturing its effect in pixel coordinates in the image plane. In our approach, the rendering model of 3DGS is decomposed into two stages: first, transforming the Gaussian parameters from world to pixel coordinates, and then rasterizing these parameters onto the image.
• Figure 4: Real data examples with scenes captured with smartphones. From top to bottom: Using COLMAP poses without motion blur compensation (baseline); with motion blur compensation (ours); reference evaluation image; $l_2$ error contributions, ours (red: 30%, yellow: 60%, white: 10%); $l_2$ error differences ours vs baseline (red: more error, blue: less error). Scenes: lego1 (iPhone), pots2 (iPhone), table (Pixel), bike (S20).
• Figure 5: Empirical ablations on deblurring with and without rolling shutter pose optimization on four real-world smartphone data scenes with rolling shutter effects. Our method with both motion blur and rolling shutter compensation (MBRS) gives sharper artefact-free reconstructions compared to just motion blur compensation (MB).