Stereo-Knowledge Distillation from dpMV to Dual Pixels for Light Field Video Reconstruction

TL;DR

The paper addresses the challenge of accurate disparity estimation from dual-pixel sensors, which offer defocus-based cues but underperform in focused regions. It proposes dark knowledge distillation from a synthetic stereo teacher to a lightweight dual-pixel student, enabling dense disparity estimation without additional hardware. A novel LF video reconstruction framework powered by vision transformers—coupled with an ensemble of geometry and flow teachers—and the dpMV dataset (first large 3-view dp video corpus) deliver the fastest and most temporally consistent LF reconstruction to date, with strong zero-shot transfer and parameter efficiency. The work enables practical, mobile-friendly 3D reconstruction and XR-ready light-field synthesis from a single dp view, highlighting substantial impact for real-world 3D vision tasks.

Abstract

Dual pixels contain disparity cues arising from the defocus blur. This disparity information is useful for many vision tasks ranging from autonomous driving to 3D creative realism. However, directly estimating disparity from dual pixels is less accurate. This work hypothesizes that distilling high-precision dark stereo knowledge, implicitly or explicitly, to efficient dual-pixel student networks enables faithful reconstructions. This dark knowledge distillation should also alleviate stereo-synchronization setup and calibration costs while dramatically increasing parameter and inference time efficiency. We collect the first and largest 3-view dual-pixel video dataset, dpMV, to validate our explicit dark knowledge distillation hypothesis. We show that these methods outperform purely monocular solutions, especially in challenging foreground-background separation regions using faithful guidance from dual pixels. Finally, we demonstrate an unconventional use case unlocked by dpMV and implicit dark knowledge distillation from an ensemble of teachers for Light Field (LF) video reconstruction. Our LF video reconstruction method is the fastest and most temporally consistent to date. It remains competitive in reconstruction fidelity while offering many other essential properties like high parameter efficiency, implicit disocclusion handling, zero-shot cross-dataset transfer, geometrically consistent inference on higher spatial-angular resolutions, and adaptive baseline control. All source code is available at the anonymous repository https://github.com/Aryan-Garg.

Figures (12)

• Figure 1: dpMV Dataset. Capture rig and a single sample.
• Figure 2: Scenes of dpMV with computed stereo-disparity. Top row: indoor scenes. Bottom row: outdoor scenes
• Figure 3: Scanline histogram analysis of dual pixels for defocus cues in foreground-background separation regions. Shown dp-disparity is from our best-fidelity method.
• Figure 4: Hypothesis Validation. Dark knowledge distilled dp-estimators outperform monocular and purely dp methods. Almost all-in-focus outdoor dpMV scene is chosen.
• Figure 5: Dark Knowledge Stereo Disparity Teacher Selection.$FT$ stands for hyper-parameter fine-tuned. The stereo teacher selected here (Unimatch xu2023unifying_unimatch) is also used as the geometry teacher for our light field video reconstruction method (\ref{['sec:lfvr']})