Two-Stream Interactive Joint Learning of Scene Parsing and Geometric Vision Tasks

TL;DR

TwInS introduces a general two-stream interactive joint learning framework that jointly tackles scene parsing and geometric vision tasks by enabling bidirectional information flow between a scene parsing stream and a geometric vision stream. It combines context-enriched iterative refinement with a cross-task adapter to fuse geometric cues into parsing and to project parsing features into geometry-informed representations, paired with a semi-supervised training strategy using uncertainty-aware pseudo labels. Empirical results on synthetic and real datasets show state-of-the-art gains in both semantic/instance segmentation and stereo/optical flow accuracy, with notable improvements over prior joint learning and feature-fusion approaches, and robust performance across different backbone configurations. The approach demonstrates versatility, compatibility with existing segmentation networks, and potential for future open-set, self-evolving perception systems with reduced annotation requirements.

Abstract

Inspired by the human visual system, which operates on two parallel yet interactive streams for contextual and spatial understanding, this article presents Two Interactive Streams (TwInS), a novel bio-inspired joint learning framework capable of simultaneously performing scene parsing and geometric vision tasks. TwInS adopts a unified, general-purpose architecture in which multi-level contextual features from the scene parsing stream are infused into the geometric vision stream to guide its iterative refinement. In the reverse direction, decoded geometric features are projected into the contextual feature space for selective heterogeneous feature fusion via a novel cross-task adapter, which leverages rich cross-view geometric cues to enhance scene parsing. To eliminate the dependence on costly human-annotated correspondence ground truth, TwInS is further equipped with a tailored semi-supervised training strategy, which unleashes the potential of large-scale multi-view data and enables continuous self-evolution without requiring ground-truth correspondences. Extensive experiments conducted on three public datasets validate the effectiveness of TwInS's core components and demonstrate its superior performance over existing state-of-the-art approaches. The source code will be made publicly available upon publication.

Figures (8)

• Figure 1: Comparison of joint learning frameworks: (a) traditional task-independent decoder design, where a shared encoder extracts visual features that are independently processed by task-specific decoders; (b) cascaded joint learning design, where disparities estimated by an iterative stereo matching network are fed into a feature-fusion semantic segmentation network via an additional spatial encoder; (c) the proposed general-purpose two-stream interactive framework, which enables bidirectional interaction by leveraging task-specific features to jointly optimize scene parsing and geometric vision tasks.
• Figure 2: An illustration of the proposed TwInS framework A pair of stereo images or consecutive video frames are fed into the framework. Contextual features extracted by the scene parsing encoder are used to guide context-enriched iterative refinement in the geometric vision stream. The iteratively refined GRU hidden states are subsequently fed back into the scene parsing stream via a cross-task adapter which provides complementary geometric cues to enhance the scene parsing performance.
• Figure 3: Qualitative results of SoTA semantic segmentation networks on the vKITTI2, Cityscapes, and KITTI 2015 datasets.
• Figure 4: Qualitative results of SoTA instance segmentation networks on the vKITTI2, Cityscapes, and KITTI 2015 datasets.
• Figure 4: Ablation study on the feature fusion strategy conducted on the Cityscapes and KITTI 2015 datasets. The proposed cross-task adapter is compared with two SoTA feature fusion strategies that incorporate cross-view geometric cues into the scene parsing stream.