BREATH-VL: Vision-Language-Guided 6-DoF Bronchoscopy Localization via Semantic-Geometric Fusion

TL;DR

This work addresses accurate 6-DoF bronchoscopy localization in challenging in-vivo airways by marrying vision-language semantics with geometry-driven registration. It introduces the BREATH dataset to enable localization-focused VLM research in surgery and proposes BREATH-VL, a dual-thread framework with a semantic initializer (BREA-VL) and a fast geometric refinement (FAM), augmented by a motion-context prompt for temporal reasoning. The approach yields robust, cross-patient localization with significant translational accuracy gains over vision-only baselines, while maintaining practical latency. The dataset, methodological contributions, and empirical results collectively advance clinically deployable, semantically-guided bronchoscopy navigation.

Abstract

Vision-language models (VLMs) have recently shown remarkable performance in navigation and localization tasks by leveraging large-scale pretraining for semantic understanding. However, applying VLMs to 6-DoF endoscopic camera localization presents several challenges: 1) the lack of large-scale, high-quality, densely annotated, and localization-oriented vision-language datasets in real-world medical settings; 2) limited capability for fine-grained pose regression; and 3) high computational latency when extracting temporal features from past frames. To address these issues, we first construct BREATH dataset, the largest in-vivo endoscopic localization dataset to date, collected in the complex human airway. Building on this dataset, we propose BREATH-VL, a hybrid framework that integrates semantic cues from VLMs with geometric information from vision-based registration methods for accurate 6-DoF pose estimation. Our motivation lies in the complementary strengths of both approaches: VLMs offer generalizable semantic understanding, while registration methods provide precise geometric alignment. To further enhance the VLM's ability to capture temporal context, we introduce a lightweight context-learning mechanism that encodes motion history as linguistic prompts, enabling efficient temporal reasoning without expensive video-level computation. Extensive experiments demonstrate that the vision-language module delivers robust semantic localization in challenging surgical scenes. Building on this, our BREATH-VL outperforms state-of-the-art vision-only localization methods in both accuracy and generalization, reducing translational error by 25.5% compared with the best-performing baseline, while achieving competitive computational latency.

Figures (10)

• Figure 1: Clinical workflow of visually-assisted bronchoscopy navigation. During robotic or conventional bronchoscopy, the navigation system receives endoscopic video frames and estimates the 6-DoF pose of the endoscope, which is then used to provide visual feedback to the surgeon. (a) Bronchoscopic frame. (b) Virtual bronchoscopy view rendered at the estimated pose. (c) Global airway view showing the endoscope’s position within the patient-specific airway.
• Figure 2: Challenging frames from the BREATH dataset. (a)-(b) show visual artifacts such as fluids and bubbles occluding the field of view. (c)-(d) show motion blur caused by rapid bronchoscope motion. (e)-(f) show textureless regions. (g)-(h) show illumination disturbances including high contrast and darkness. (i)-(j) show anatomically distinct airway regions with similar visual appearance, which can confuse landmark-based methods; (i) is from the left inferior lobar bronchus, and (j) is from the right intermediate bronchus.
• Figure 3: Overview of BREATH-VL for 6-DoF bronchoscopy localization. At time $t$, the bronchoscope pose $s_t$ is initialized with a semantic prior from BREA-VL and then refined via registration using vision-only geometric features.
• Figure 4: Overview of FAM for fine-grained bronchoscope localization. At time $t$, the bronchoscope pose $s_t$ is estimated by optimizing a composite objective that combines depth similarity, landmark alignment, and a centerline constraint. Localization accuracy is further improved through interaction with BREA-VL, which provides semantic pose initialization and landmark consistency checking.
• Figure 5: Data annotation pipeline for the BREATH dataset. After segmentation, 3D reconstruction, and anatomical mapping of the patient-specific airway, we first manually annotate the 6-DoF bronchoscope pose by registering virtual bronchoscopy images to real images. Using the labeled airway centerline, we then automatically generate semantic labels, including branch-level localization, insertion depth, and landmark detection. Finally, we convert these semantic labels into VQA annotations to build BREATH-VL.