SHED Light on Segmentation for Dense Prediction

TL;DR

SHED addresses structural incoherence in dense prediction by learning a bidirectional segment hierarchy within a ViT-based encoder–decoder. It builds a forward segment hierarchy in the encoder and reverses it in the decoder to constrain pixel-level outputs with global scene geometry, all learned end-to-end under the dense prediction objective. Empirically, SHED achieves sharper depth boundaries, better intra-segment consistency, and strong cross-domain generalization, while enabling 3D reconstructions with interpretable 3D parts. This approach advances geometry-aware perception for robotics and autonomous systems by coupling depth, segmentation, and 3D structure in a single architecture.

Abstract

Dense prediction infers per-pixel values from a single image and is fundamental to 3D perception and robotics. Although real-world scenes exhibit strong structure, existing methods treat it as an independent pixel-wise prediction, often resulting in structural inconsistencies. We propose SHED, a novel encoder-decoder architecture that enforces geometric prior explicitly by incorporating segmentation into dense prediction. By bidirectional hierarchical reasoning, segment tokens are hierarchically pooled in the encoder and unpooled in the decoder to reverse the hierarchy. The model is supervised only at the final output, allowing the segment hierarchy to emerge without explicit segmentation supervision. SHED improves depth boundary sharpness and segment coherence, while demonstrating strong cross-domain generalization from synthetic to the real-world environments. Its hierarchy-aware decoder better captures global 3D scene layouts, leading to improved semantic segmentation performance. Moreover, SHED enhances 3D reconstruction quality and reveals interpretable part-level structures that are often missed by conventional pixel-wise methods.

Figures (14)

• Figure 1: Segment hierarchy for dense prediction (SHED). Conventional methods such as DPT ranftl2021vision perform pixel-wise prediction without considering structure, often resulting in blurry object shapes. SHED addresses this by leveraging a hierarchy of segment tokens to guide prediction. Unlike DPT, which uses fixed grid tokens across all layers, we adapt its ViT dosovitskiy2020image blocks into two stages: the encoder pools superpixel tokens into coarser segment tokens, and the decoder progressively refines predictions from coarse to fine segments, producing depth maps with structural coherence.
• Figure 2: SHED introduces a bidirectional segment hierarchy for dense prediction. SHED decomposes dense prediction into two complementary processes: structural abstraction and structural inversion. 1) The encoder constructs a forward hierarchy by grouping superpixel tokens into increasingly abstract segments. 2) The decoder explicitly inverts this hierarchy by unpooling segment representations from coarse to fine, enabling global structural context to directly constrain pixel-level predictions. Segment tokens are projected into region-aligned 2D feature maps and fused across hierarchy levels, together with early convolutional features, to recover fine details and produce the final dense map.
• Figure 3: SHED produces consistent structures in predicted depth map with spatio-layout. We visualize the fine-to-coarse segments and corresponding depth maps from SHED, along with ground truth (GT) depth. Examples are from the NYUv2 test set. SHED captures fine structures through its segments, such as desks in a classroom, which allow the depth map to clearly separate them from the background (row 1). It also decomposes large objects, such as a table, into multiple parts, leading to smooth depth variations toward the back (row 2).
• Figure 4: SHED generates sharper object contours, clearer occlusion boundaries, and more coherent values within segments. We compare depth maps (cols 2-4) and occlusion boundaries (cols 5, 6) from DPT, SHED on the NYUv2-OC++ dataset. Boundaries are extracted using a Canny edge detector and evaluated against GT, with GT edges shown in yellow, true positive in green and false positive in red. SHED more accurately captures object edges and produces smoother depth within segments. Its predicted boundaries also align more closely with the ground truth.
• Figure 4: 3D alignment induced by structured depth estimation. We compute the Chamfer distance between point clouds from the predicted and ground-truth depths. SHED achieves lower errors than DPT, indicating improved geometric alignment.