Guardians of the Hair: Rescuing Soft Boundaries in Depth, Stereo, and Novel Views

TL;DR

HairGuard tackles the persistent challenge of soft boundaries in 3D vision by leveraging image matting data to train a depth fixer with a gated residual that precisely refines depth at soft boundaries, while preserving global depth quality. In view synthesis, a forward warping step using the corrected depth is followed by a scene painter to fill disoccluded regions and a color fuser to blend warped and inpainted textures, ensuring coherent geometry and high-fidelity details. The approach yields state-of-the-art results across monocular depth estimation, stereo image/video conversion, and novel view synthesis, with pronounced gains in soft-boundary regions and robust zero-shot performance. The framework is modular and plug-and-play with existing depth models, making it practical for real-world 3D vision pipelines involving hair-like and delicate structures.

Abstract

Soft boundaries, like thin hairs, are commonly observed in natural and computer-generated imagery, but they remain challenging for 3D vision due to the ambiguous mixing of foreground and background cues. This paper introduces Guardians of the Hair (HairGuard), a framework designed to recover fine-grained soft boundary details in 3D vision tasks. Specifically, we first propose a novel data curation pipeline that leverages image matting datasets for training and design a depth fixer network to automatically identify soft boundary regions. With a gated residual module, the depth fixer refines depth precisely around soft boundaries while maintaining global depth quality, allowing plug-and-play integration with state-of-the-art depth models. For view synthesis, we perform depth-based forward warping to retain high-fidelity textures, followed by a generative scene painter that fills disoccluded regions and eliminates redundant background artifacts within soft boundaries. Finally, a color fuser adaptively combines warped and inpainted results to produce novel views with consistent geometry and fine-grained details. Extensive experiments demonstrate that HairGuard achieves state-of-the-art performance across monocular depth estimation, stereo image/video conversion, and novel view synthesis, with significant improvements in soft boundary regions.

Figures (19)

• Figure 1: Our Mission. The Guardians of the Hair (HairGuard) aim to rescue soft boundary details, e.g., thin hairs, where foreground and background are mixed in the observed color. Previous state-of-the-art approaches often suffer from missing details (see depth estimation results), degraded texture (see stereo results, displayed in anaglyph), and inconsistent geometry (see novel views) in soft boundaries. In contrast, HairGuard preserves fine-grained soft boundary details and demonstrates strong performance across diverse tasks.
• Figure 2: Soft boundaries. Existing depth estimation methods often struggle to capture accurate depth in soft boundaries, resulting in discontinuous depth (red box) and broken boundaries (green box). DAv2, DPro, and UDv2 represent Depth Anything V2 yang2024depthanythingv2, Depth Pro depthpro, and UniDepthV2 unidepthv2, respectively.
• Figure 3: HairGuard pipeline. Given an input image and its estimated depth, we first design a depth fixer to refine depth predictions around soft boundary regions. The fixed depth is then used for forward warping to generate preliminary novel views, which are fed into the scene painter for disocclusion inpainting. Finally, our color fuser adaptively combines the warped and inpainted results to produce geometrically and visually consistent novel views.
• Figure 4: Depth fixer. (a) We utilize image matting datasets to synthesize training data with fine-grained depth labels in soft boundaries. (b) Instead of relying on manually crafted cues like trimaps yao2024vitmatte, we leverage depth maps and image semantics to automatically identify soft boundary regions. The gated residual module enables precise depth correction in soft boundary areas and thus benefits plug-and-play refinement. (c) Compared with direct prediction and vanilla residual, our gated residual combined with two-stage training achieves the best depth results.
• Figure 5: Color fuser. (a) We employ image matting datasets and multi-view datasets to synthesize warped and inpainted results for training. (b) Built upon a pre-trained VAE, we design a dual skip module to leverage the merits of inapinted and warped images. (c) Our color fuser eliminates redundant background colors (green box in the warped image) and hallucinated textures (red box in the inpainted image) for high-quality view synthesis.