The Forgotten Shield: Safety Grafting in Parameter-Space for Medical MLLMs

TL;DR

This paper addresses safety gaps in Medical MLLMs, showing susceptibility to cross-modality jailbreaks and catastrophic forgetting of safety during medical fine-tuning. It introduces Parameter-Space Intervention (PSI), which extracts a safety vector from a base model, orthogonally disentangles it from the medical capability via Gram-Schmidt, and performs fine-grained layer-wise CMA-ES optimization to maximize a joint objective $R(\theta) = \lambda_1 S_{med} + \lambda_2 S_{safe}$. Empirical results on 7B-scale Medical and general MLLMs demonstrate substantial gains in safety across both general and medical-specific benchmarks with minimal degradation to medical performance, outperforming ModelMerge and RESTA baselines. The approach requires no additional domain-specific safety data, offering a cost-efficient, scalable path toward deploying safe Medical MLLMs, while acknowledging limitations in scalability to much larger models and vector derivation strategies for safety representations.

Abstract

Medical Multimodal Large Language Models (Medical MLLMs) have achieved remarkable progress in specialized medical tasks; however, research into their safety has lagged, posing potential risks for real-world deployment. In this paper, we first establish a multidimensional evaluation framework to systematically benchmark the safety of current SOTA Medical MLLMs. Our empirical analysis reveals pervasive vulnerabilities across both general and medical-specific safety dimensions in existing models, particularly highlighting their fragility against cross-modality jailbreak attacks. Furthermore, we find that the medical fine-tuning process frequently induces catastrophic forgetting of the model's original safety alignment. To address this challenge, we propose a novel "Parameter-Space Intervention" approach for efficient safety re-alignment. This method extracts intrinsic safety knowledge representations from original base models and concurrently injects them into the target model during the construction of medical capabilities. Additionally, we design a fine-grained parameter search algorithm to achieve an optimal trade-off between safety and medical performance. Experimental results demonstrate that our approach significantly bolsters the safety guardrails of Medical MLLMs without relying on additional domain-specific safety data, while minimizing degradation to core medical performance.

Figures (2)

• Figure 1: The safety dilemma in current Medical MLLMs. Despite achieving remarkable performance on specialized medical tasks (top-left), SOTA models exhibit severe vulnerabilities in both general (top-right) and medical-specific (bottom) safety dimensions. Our empirical analysis, visualized in the central radar chart, reveals that medical fine-tuning frequently induces catastrophic forgetting of the base model's original safety alignment, particularly against cross-modality jailbreak attacks.
• Figure 2: Overview of the proposed Parameter-Space Intervention framework. (Left) This panel illustrates the vector extraction and disentanglement process: (1) Safety Vector is derived by contrasting the base model (e.g., Qwen2.5-VL) with an unaligned version; (2) a Medical Performance Vector is obtained from the medically fine-tuned model (e.g., Lingshu); and (3) Gram-Schmidt Orthogonalization is applied to decouple the safety vector from the medical performance vector, preventing destructive interference. (Middle) This panel visualizes the fine-grained optimization stage, where a search algorithm (CMA-ES) explores the high-dimensional parameter space to find optimal layer-wise coefficients ($\alpha^{(l)}, \beta^{(l)}$) that balance the Medical Performance and Safety. (Right) This panel provides a qualitative comparison demonstrating the re-alignment's effectiveness. The original fine-tuned model (red boxes) generates harmful content for both general and medical-specific safety attacks, while our re-aligned model (blue boxes) robustly refuses these requests, successfully restoring safety guardrails