Medical Vision Language Models as Policies for Robotic Surgery

TL;DR

This work tackles the difficulty of learning from endoscopic visuals in robotic laparoscopic RL by injecting domain-specific planning through MedFlamingo PPO. A frozen 7B medical Vision-Language backbone generates high-level planning tokens $m_t ∈ \mathbb{R}^k$ from the initial observation and task description, which are fused with ResNet visual features to form $s_t$ for PPO updates, with tokens produced once per episode. Across five LapGym environments—spanning spatial reasoning, deformable tissue manipulation, dissection, threading, and grasping—the approach achieves task success $>70\%$ in all tasks and shows large relative improvements over baselines (e.g., up to $1114.29\%$ in DeflectSpheresEnv). The results demonstrate the value of incorporating domain-specific medical knowledge into robotic surgical planning, enabling faster convergence and improved robustness using endoscopic observations, with clear implications for real-world autonomous surgical systems.

Abstract

Vision-based Proximal Policy Optimization (PPO) struggles with visual observation-based robotic laparoscopic surgical tasks due to the high-dimensional nature of visual input, the sparsity of rewards in surgical environments, and the difficulty of extracting task-relevant features from raw visual data. We introduce a simple approach integrating MedFlamingo, a medical domain-specific Vision-Language Model, with PPO. Our method is evaluated on five diverse laparoscopic surgery task environments in LapGym, using only endoscopic visual observations. MedFlamingo PPO outperforms and converges faster compared to both standard vision-based PPO and OpenFlamingo PPO baselines, achieving task success rates exceeding 70% across all environments, with improvements ranging from 66.67% to 1114.29% compared to baseline. By processing task observations and instructions once per episode to generate high-level planning tokens, our method efficiently combines medical expertise with real-time visual feedback. Our results highlight the value of specialized medical knowledge in robotic surgical planning and decision-making.

Figures (4)

• Figure 1: MedFlamingo processes the initial observation and task description to generate planning tokens. These tokens can be concatenated with ResNet-encoded visual features at each timestep.
• Figure 2: Illustration of the PPO agent's interaction with the LapGym environment. At each time step $t$, the agent observes the environment state and uses its Critic and Actor networks to produce a value estimate $v_t$ and an action $a_t$, respectively. The action $a_t$ is applied to the environment, which responds with the next state and a reward signal. This reward is used to update both the Actor and Critic networks through the PPO update mechanism. The environment state representation combines ResNet-encoded visual features with optional task-relevant planning tokens: MedFlamingo tokens (domain-specific), OpenFlamingo tokens (general-purpose), or no tokens (Baseline PPO).
• Figure 3: Performance comparison (10 seed runs for each method) of MedFlamingo PPO, OpenFlamingo PPO, and Baseline PPO on the RopeCuttingEnv, TissueManipulationEnv, DeflectSpheresEnv, RopeThreadingEnv, and PickAndPlaceEnv. Success rates are reported on test environment seeds and normalized over total success rates.
• Figure 4: Reward-weighted state visitation heatmaps for MedFlamingo PPO, OpenFlamingo PPO, and Baseline PPO in RopeThreadingEnv. Clustered states capture fine-grained exploration patterns, with visitation frequencies weighted by rewards representing task-relevant regions. MedFlamingo PPO demonstrates efficient, targeted exploration focused on high-reward regions, while OpenFlamingo PPO shows moderate focus with some spillover into less relevant states. Baseline PPO exhibits scattered and inefficient exploration, with sparse activity in key areas.