Mamba as a Bridge: Where Vision Foundation Models Meet Vision Language Models for Domain-Generalized Semantic Segmentation

TL;DR

This work tackles domain-generalized semantic segmentation by integrating Vision Foundation Models (VFMs) and Vision-Language Models (VLMs) through a lightweight, scalable fusion framework. The authors introduce MVFuser, a Mamba-based co-adapter that jointly refines VFM and VLM features, and MTEnhancer, a hybrid attention-Mamba module that enriches text embeddings with visual priors, preserving the encoders' generalization by keeping them frozen. Trained only on the proposed adapters and a Mask2Former decoder, MFuser achieves state-of-the-art results on synthetic-to-real and real-to-real DGSS benchmarks, with reported mIoU of 68.20 and 71.87 respectively, while maintaining linear complexity in sequence length. The approach demonstrates that carefully designed cross-modal adapters can unlock the complementary strengths of VFMs and VLMs, enabling robust, domain-appropriate segmentation without prohibitive computational costs. This has practical implications for deploying DGSS models in real-world, varied environments such as autonomous driving.

Abstract

Vision Foundation Models (VFMs) and Vision-Language Models (VLMs) have gained traction in Domain Generalized Semantic Segmentation (DGSS) due to their strong generalization capabilities. However, existing DGSS methods often rely exclusively on either VFMs or VLMs, overlooking their complementary strengths. VFMs (e.g., DINOv2) excel at capturing fine-grained features, while VLMs (e.g., CLIP) provide robust text alignment but struggle with coarse granularity. Despite their complementary strengths, effectively integrating VFMs and VLMs with attention mechanisms is challenging, as the increased patch tokens complicate long-sequence modeling. To address this, we propose MFuser, a novel Mamba-based fusion framework that efficiently combines the strengths of VFMs and VLMs while maintaining linear scalability in sequence length. MFuser consists of two key components: MVFuser, which acts as a co-adapter to jointly fine-tune the two models by capturing both sequential and spatial dynamics; and MTEnhancer, a hybrid attention-Mamba module that refines text embeddings by incorporating image priors. Our approach achieves precise feature locality and strong text alignment without incurring significant computational overhead. Extensive experiments demonstrate that MFuser significantly outperforms state-of-the-art DGSS methods, achieving 68.20 mIoU on synthetic-to-real and 71.87 mIoU on real-to-real benchmarks. The code is available at https://github.com/devinxzhang/MFuser.

Figures (6)

• Figure 1: Comparative analysis of the VFM and the VLM features. VFM: Visualization of PCA-computed features from DINOv2 (the first three components of PCA, computed on the image features, serve as color channels), displaying fine-grained details but lacking text alignment. VLM: Image-text similarity map from EVA02-CLIP using the query 'car', demonstrating good alignment with text but insufficient localization of queried objects. MFuser: Our proposed fusion framework integrates VFM and VLM, resulting in unified features that exhibit both precise locality and robust text alignment. Quantitative results on synthetic-to-real DGSS benchmarks further validate our approach, with MFuser consistently achieving the highest mIoU scores across all tasks.
• Figure 2: Overall architecture of MFuser. MFuser takes inputs through both VFM and VLM visual encoders. Features from each encoder layer are concatenated and refined in MVFuser, which captures sequential and spatial dependencies in parallel. The refined features are then added back to the original features and passed to the next layer. MTEnhancer strengthens text embeddings of each class by integrating visual features through a hybrid attention-Mamba mechanism. The enhanced text embeddings serve as object queries for the Mask2Former decoder, alongside multi-scale visual features. During training, only MVFusers, MTEnhancers, and the segmentation decoder are trainable while the VFM and VLM remain frozen, preserving their generalization ability and enabling efficient training. Note that skip connections between each block of MTEnhancer are omitted for clarity.
• Figure 3: Qualitative results on unseen target domains under the G$\rightarrow${C, B, M} setting. MFuser is compared with Rein wei2024stronger and tqdm pak2025textual.
• Figure 4: PCA visualization of features from DINOv2 and EVA02-CLIP, illustrating how MVFuser-based adaptation refines their distributions before and after tuning.
• Figure 5: Qualitative results on unseen target domains under the G$\rightarrow$M setting. MFuser is compared with Rein wei2024stronger and tqdm pak2025textual.