Bridging the Perception-Cognition Gap:Re-engineering SAM2 with Hilbert-Mamba for Robust VLM-based Medical Diagnosis

TL;DR

The paper tackles the difficulty of applying general Vision-Language Models to 3D multimodal medical imaging, where subtle lesions and complex spatial relationships impede reliable diagnosis. It introduces Hilbert-VLM, a two-stage framework that first performs precise lesion segmentation with HilbertMed-SAM (a re-engineered SAM2 using Hilbert space-filling curves and memory-augmented Mamba state-space models) and then converts segmentation results into enhanced multimodal prompts to drive a VLM-based classifier. Key contributions include the Hilbert-Mamba architectural design, memory-infused multimodal fusion, a dual-path decoder for precise segmentation, and a fused prompt module that aligns VLM reasoning with visual evidence, achieving state-of-the-art segmentation on BraTS2021 and FCD2023 and leading end-to-end diagnostic performance. The approach significantly improves the reliability and accuracy of AI-assisted clinical decision making in brain pathology, with strong potential for deployment in AI-enabled clinical workflows.

Abstract

Recent studies suggest that Visual Language Models (VLMs) hold great potential for tasks such as automated medical diagnosis. However, processing complex three-dimensional (3D) multimodal medical images poses significant challenges - specifically, the effective integration of complementary information and the occasional oversight of subtle yet critical pathological features. To address these issues, we present a novel two-stage fusion framework termed Hilbert-VLM. This framework leverages the HilbertMed-SAM module for precise lesion segmentation, with the generated multimodal enhanced prompts then guiding the VLM toward accurate disease classification. Our key innovation lies in the systematic redesign of the Segment Anything Model 2 (SAM2) architecture: we incorporate Hilbert space-filling curves into the scanning mechanism of the Mamba State Space Model (SSM) to maximize the preservation of spatial locality in 3D data, a property critical for medical image analysis. We also introduce a novel Hilbert-Mamba Cross-Attention (HMCA) mechanism and a scale-aware decoder to capture fine-grained details. Meanwhile, the prompt enhancement module unifies segmentation masks and their corresponding textual attributes into an information-dense prompt to support VLM inference. Extensive experiments were conducted to validate the effectiveness of the Hilbert-VLM model. On the BraTS2021 segmentation benchmark, it achieves a Dice score of 82.35 percent, with a diagnostic classification accuracy (ACC) of 78.85 percent. These results demonstrate that the proposed model offers substantial potential to improve the accuracy and reliability of medical VLM-based analysis.

Figures (5)

• Figure 1: The architecture of HilbertMed-SAM encompasses multimodal input and HMCA fusion, the Hilbert Mamba optimisation module and a dual-path (Stage 1/Stage 2) decoder for the final prediction.
• Figure 2: The Hilbert Mamba block is based on a particular scanning principle. Unlike standard grating scans, which destroy spatial locality, our approach uses a 3D Hilbert curve scanning strategy.
• Figure 3: Detailed architecture of the Hilbert Mamba Module.
• Figure 4: An illustration of the Stage 2 Fused Prompting mechanism for VLM-based classification.
• Figure 5: Qualitative comparison of segmentation results from our HilbertMed-SAM against other state-of-the-art methods on representative slices from the BraTS2021 and FCD2023 datasets.