AdaGaR: Adaptive Gabor Representation for Dynamic Scene Reconstruction

TL;DR

AdaGaR addresses dynamic scene reconstruction from monocular video by unifying frequency adaptivity with temporal continuity. It introduces Adaptive Gabor Primitives that learn frequency weights with an energy-compensation mechanism, enabling high-frequency texture capture while preserving stability. Temporal dynamics are constrained with Cubic Hermite Splines and a Temporal Curvature Regularization to ensure smooth motion, aided by an adaptive initialization strategy. On Tap-Vid DAVIS, AdaGaR achieves state-of-the-art PSNR/SSIM/LPIPS and generalizes to frame interpolation, depth consistency, video editing, and stereo view synthesis, demonstrating practical impact for high-fidelity dynamic video rendering.

Abstract

Reconstructing dynamic 3D scenes from monocular videos requires simultaneously capturing high-frequency appearance details and temporally continuous motion. Existing methods using single Gaussian primitives are limited by their low-pass filtering nature, while standard Gabor functions introduce energy instability. Moreover, lack of temporal continuity constraints often leads to motion artifacts during interpolation. We propose AdaGaR, a unified framework addressing both frequency adaptivity and temporal continuity in explicit dynamic scene modeling. We introduce Adaptive Gabor Representation, extending Gaussians through learnable frequency weights and adaptive energy compensation to balance detail capture and stability. For temporal continuity, we employ Cubic Hermite Splines with Temporal Curvature Regularization to ensure smooth motion evolution. An Adaptive Initialization mechanism combining depth estimation, point tracking, and foreground masks establishes stable point cloud distributions in early training. Experiments on Tap-Vid DAVIS demonstrate state-of-the-art performance (PSNR 35.49, SSIM 0.9433, LPIPS 0.0723) and strong generalization across frame interpolation, depth consistency, video editing, and stereo view synthesis. Project page: https://jiewenchan.github.io/AdaGaR/

Figures (14)

• Figure 1: State-of-the-art video reconstruction quality on DAVIS pont20172017 dataset. Our Adaptive Gabor representation achieves superior rendering quality (PSNR: 35.49 dB, SSIM: 0.9433) while preserving fine details and temporal consistency. (Left) Qualitative comparisons demonstrate sharper textures in challenging regions (car windows, drum surface) compared to CoDeF ouyang2024codef and Splatter A Video sun2024splatter. (Right) Our method (red point) significantly outperforms recent baselines across all metrics, achieving 6.86 dB PSNR improvement over the second-best method with reasonable training time (circle size indicates training duration: 30 mins to 24 hours).
• Figure 2: Hierarchical frequency adaptation. Our primitives adaptively transition from Gaussian (topleft) to Gabor (bottom), enabling coarse-to-fine reconstruction. Each primitive learns its optimal frequency response via learnable weights $\omega_i$, achieving both geometric stability and texture detail in a unified framework.
• Figure 3: Method overview. Our approach represents dynamic videos as Adaptive Gabor primitives with temporally smooth motion. (Input) Multi-modal supervision from RGB, depth, tracking, and masks. (Optimization) Two core components: (1) Adaptive Motion: Cubic Hermite splines model primitive trajectories with control points $\mu(t)$, $q(t)$ in orthographic camera space, ensuring C$^1$ continuity. (2) Adaptive Gabor Representation: Learnable frequency weights $\omega_k$ enable primitives to adaptively span from Gaussian (low-freq) to Gabor (high-freq), achieving hierarchical detail reconstruction. (Loss) Joint optimization via RGB, depth, flow supervision, and curvature regularization $L_{curv}$. (Application) Supports frame interpolation, depth consistency, and video editing.
• Figure 4: Adaptive Gabor formulation.(a) Smooth transition between Gaussian and Gabor kernels. Our method (rightmost column, $S_\text{ours}(x)$) uses a compensation term $b$ to maintain energy stability while transitioning from pure Gaussian ($\omega=0$, top) to frequency-modulated Gabor ($\omega=1$, bottom). Naive combination $1+S(x)$ (third column) suffers from intensity artifacts. (b) Frequency weight combinations. Different ($\omega_0$, $\omega_1$) pairs generate diverse spatial patterns, from smooth (low $\omega$) to high-frequency textures (high $\omega$), enabling adaptive detail capture in different scene regions.
• Figure 5: Visual comparison on DAVIS dataset. Our method preserves finer details (fur, vehicle edges, wheel structures) and sharper motion boundaries compared to CoDeF ouyang2024codef and Splatter A Video sun2024splatter. Red boxes highlight key regions demonstrating our superior texture reconstruction and temporal consistency. Best viewed zoomed in.