Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DINO-CVA: A Multimodal Goal-Conditioned Vision-to-Action Model for Autonomous Catheter Navigation

Pedram Fekri, Majid Roshanfar, Samuel Barbeau, Seyedfarzad Famouri, Thomas Looi, Dale Podolsky, Mehrdad Zadeh, Javad Dargahi

TL;DR

This work tackles autonomous catheter navigation by learning from expert demonstrations through a multimodal, goal-conditioned imitation-learning framework called DINO-CVA. The model jointly embeds visual scenes and kinematic joystick states using a frozen DINOv2 backbone, a cross-attention fusion, and a gated mechanism conditioned on a goal image to predict the next control signal $s_{t+1}$ in an autoregressive fashion. Experiments on a robotic phantom demonstrate high action-prediction accuracy and grounding in anatomical context, with ablations showing the necessity of both modalities and accurate goal conditioning. The approach advances toward reducing operator dependency and radiation exposure by enabling vision-aware, trajectory-aware autonomous navigation within vascular anatomies, while outlining concrete paths for improving realism and generalization."

Abstract

Cardiac catheterization remains a cornerstone of minimally invasive interventions, yet it continues to rely heavily on manual operation. Despite advances in robotic platforms, existing systems are predominantly follow-leader in nature, requiring continuous physician input and lacking intelligent autonomy. This dependency contributes to operator fatigue, more radiation exposure, and variability in procedural outcomes. This work moves towards autonomous catheter navigation by introducing DINO-CVA, a multimodal goal-conditioned behavior cloning framework. The proposed model fuses visual observations and joystick kinematics into a joint embedding space, enabling policies that are both vision-aware and kinematic-aware. Actions are predicted autoregressively from expert demonstrations, with goal conditioning guiding navigation toward specified destinations. A robotic experimental setup with a synthetic vascular phantom was designed to collect multimodal datasets and evaluate performance. Results show that DINO-CVA achieves high accuracy in predicting actions, matching the performance of a kinematics-only baseline while additionally grounding predictions in the anatomical environment. These findings establish the feasibility of multimodal, goal-conditioned architectures for catheter navigation, representing an important step toward reducing operator dependency and improving the reliability of catheterbased therapies.

DINO-CVA: A Multimodal Goal-Conditioned Vision-to-Action Model for Autonomous Catheter Navigation

TL;DR

This work tackles autonomous catheter navigation by learning from expert demonstrations through a multimodal, goal-conditioned imitation-learning framework called DINO-CVA. The model jointly embeds visual scenes and kinematic joystick states using a frozen DINOv2 backbone, a cross-attention fusion, and a gated mechanism conditioned on a goal image to predict the next control signal in an autoregressive fashion. Experiments on a robotic phantom demonstrate high action-prediction accuracy and grounding in anatomical context, with ablations showing the necessity of both modalities and accurate goal conditioning. The approach advances toward reducing operator dependency and radiation exposure by enabling vision-aware, trajectory-aware autonomous navigation within vascular anatomies, while outlining concrete paths for improving realism and generalization."

Abstract

Cardiac catheterization remains a cornerstone of minimally invasive interventions, yet it continues to rely heavily on manual operation. Despite advances in robotic platforms, existing systems are predominantly follow-leader in nature, requiring continuous physician input and lacking intelligent autonomy. This dependency contributes to operator fatigue, more radiation exposure, and variability in procedural outcomes. This work moves towards autonomous catheter navigation by introducing DINO-CVA, a multimodal goal-conditioned behavior cloning framework. The proposed model fuses visual observations and joystick kinematics into a joint embedding space, enabling policies that are both vision-aware and kinematic-aware. Actions are predicted autoregressively from expert demonstrations, with goal conditioning guiding navigation toward specified destinations. A robotic experimental setup with a synthetic vascular phantom was designed to collect multimodal datasets and evaluate performance. Results show that DINO-CVA achieves high accuracy in predicting actions, matching the performance of a kinematics-only baseline while additionally grounding predictions in the anatomical environment. These findings establish the feasibility of multimodal, goal-conditioned architectures for catheter navigation, representing an important step toward reducing operator dependency and improving the reliability of catheterbased therapies.
Paper Structure (18 sections, 7 equations, 6 figures, 3 tables)

This paper contains 18 sections, 7 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reinforcement Learning (RL)
  4. Imitation Learning (IL)
  5. Inverse Reinforcement Learning (IRL)
  6. Challenges of Existing Paradigms
  7. Experimental Setup and Data Compilation
  8. Robotic Platform for Catheter Navigation
  9. Interface Software
  10. Data Compilation
  11. Cath-VA: A Vision-Action model for Catheter Navigation
  12. Methodology
  13. Results and Discussions
  14. Data Preparation and Model Configuration
  15. Results
  16. ...and 3 more sections

Figures (6)

  • Figure 1: Overall experimental setup. The system includes a transparent vessel phantom placed under a top-mounted camera, a robotic platform with an actuator unit mounted on a linear rail, and a joystick controller for teleoperation. A buckle-prevention guide ensures stable catheter insertion. The graphical user interface allows parameter configuration and provides real-time visualization of the catheter within the phantom.
  • Figure 2: Close-up of the robotic platform. Three stepper motors, each driven by a microstepping driver, provide three DoFs: catheter translation, distal tip steering, and base rotation. The system is mounted on an optical breadboard, with 3D-printed fixtures securing the catheter during operation.
  • Figure 3: This figure illustrates the architecture of DINO-CVA. The model receives video frames of the catheterization task along with the motors’ states (joystick inputs). The frames are encoded using a frozen DINOv2 backbone and buffered to form a temporal sequence. The motors’ states are linearly projected to match the visual embedding space. After positional embeddings are applied, cross-attention fuses the two modalities, while spatial and temporal dependencies are modeled by a causal transformer. The fused representation is then conditioned on the target frame through a gated fusion network, and the action head predicts the next state $s_{t+1}$.
  • Figure 4: The figure depicts nine predefined starting and ending points, indicated by red dots on the artificial vasculature. These images also represent sample visual data used to train and evaluate DINO-CVA. The starting point was kept identical across all experiments, while the target endpoints varied among the nine locations.
  • Figure 5: Violin plots illustrating the empirical distribution of motor states across both dataset versions (episode-based and scenario-based) for training, validation, and test splits. The plots highlight the variability in catheter control inputs and their corresponding motion trajectories.
  • ...and 1 more figures