DepthFocus: Controllable Depth Estimation for See-Through Scenes

TL;DR

DepthFocus tackles depth ambiguity in see-through scenes by enabling intent-driven control over depth perception with a scalar conditioning variable $c\in[0,1]$ in a stereo setting. It introduces a steerable Vision Transformer with conditional Mixture-of-Experts routing and direct condition injection, built on the S²M² backbone, and trained on a synthetic multi-layer dataset of about $5\times 10^5$ pairs. The method achieves state-of-the-art performance on traditional single-depth benchmarks like BOOSTER and demonstrates controllable, multi-layer depth estimation in synthetic and real-world data, including LayeredFlow, with $d_{ref}=(1-c)d_{max}$ used for supervision. This work advances active, human-like 3D perception by enabling selective focus on depths of interest in complex see-through environments and providing new synthetic/real benchmarks for multi-layer depth estimation.

Abstract

Depth in the real world is rarely singular. Transmissive materials create layered ambiguities that confound conventional perception systems. Existing models remain passive, attempting to estimate static depth maps anchored to the nearest surface, while humans actively shift focus to perceive a desired depth. We introduce DepthFocus, a steerable Vision Transformer that redefines stereo depth estimation as intent-driven control. Conditioned on a scalar depth preference, the model dynamically adapts its computation to focus on the intended depth, enabling selective perception within complex scenes. The training primarily leverages our newly constructed 500k multi-layered synthetic dataset, designed to capture diverse see-through effects. DepthFocus not only achieves state-of-the-art performance on conventional single-depth benchmarks like BOOSTER, a dataset notably rich in transparent and reflective objects, but also quantitatively demonstrates intent-aligned estimation on our newly proposed real and synthetic multi-depth datasets. Moreover, it exhibits strong generalization capabilities on unseen see-through scenes, underscoring its robustness as a significant step toward active and human-like 3D perception.

Figures (5)

• Figure 1: Strong generalization on the LayeredFlow dataset wen2024layeredflow demonstrates that DepthFocus effectively handles multi-layered depth environments. By adapting to user intent, our model aligns depth in transmissive regions and adjusts layers according to distance control $c$.
• Figure 2: Overview of DepthFocus architecture. The model builds on a multi-resolution transformer backbone for global stereo matching, augmented with conditional modules that enable controllable depth adjustment. The right panel illustrates our two conditional operation modules.
• Figure 3: Our multi-layered synthetic dataset. Top row: diverse environments with generated multi-layered depth. Bottom row: annotations capturing depth ambiguities from transmissive and reflective materials.
• Figure 4: Comparison on the Booster dataset ramirez2022open, which contains many transmissive/reflective regions. DepthFocus provides accurate front-surface and reliable through-view results, while other methods yield inconsistent predictions.
• Figure 5: Sample scenes from our real test dataset with varying transmissivity: (a) no plate, (b) 80$\%$ transmissivity, and (c) 60$\%$ transmissivity.