Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AM$^3$Safety: Towards Data Efficient Alignment of Multi-modal Multi-turn Safety for MLLMs

Han Zhu, Jiale Chen, Chengkun Cai, Shengjie Sun, Haoran Li, Yujin Zhou, Chi-Min Chan, Pengcheng Wen, Lei Li, Sirui Han, Yike Guo

TL;DR

AM$^3$Safety advances safety alignment for multi-turn multimodal LLMs by introducing InterSafe-V, a large-scale dialogue dataset with specialized refusal VQA samples, and a data-efficient training framework that couples a cold-start refusal phase with GRPO-based fine-tuning using a turn-aware dual-objective reward. The approach uses safety variance-based turn weighting to identify critical turns and balances helpfulness with safety via a dual reward, achieving over 10% ASR reduction and at least 8% improvement in harmlessness and over 13% in helpfulness on multi-turn benchmarks, while preserving general capabilities. The work emphasizes model-to-model data generation to reduce manual annotation costs and demonstrates strong performance on Qwen2.5-VL-7B-Instruct and LLaVA-NeXT-7B across SafeMT, JailbreakV, and MMSafe-PO, offering a practical, scalable path for safety in real-world multimodal dialogues.

Abstract

Multi-modal Large Language Models (MLLMs) are increasingly deployed in interactive applications. However, their safety vulnerabilities become pronounced in multi-turn multi-modal scenarios, where harmful intent can be gradually reconstructed across turns, and security protocols fade into oblivion as the conversation progresses. Existing Reinforcement Learning from Human Feedback (RLHF) alignment methods are largely developed for single-turn visual question-answer (VQA) task and often require costly manual preference annotations, limiting their effectiveness and scalability in dialogues. To address this challenge, we present InterSafe-V, an open-source multi-modal dialogue dataset containing 11,270 dialogues and 500 specially designed refusal VQA samples. This dataset, constructed through interaction between several models, is designed to more accurately reflect real-world scenarios and includes specialized VQA pairs tailored for specific domains. Building on this dataset, we propose AM$^3$Safety, a framework that combines a cold-start refusal phase with Group Relative Policy Optimization (GRPO) fine-tuning using turn-aware dual-objective rewards across entire dialogues. Experiments on Qwen2.5-VL-7B-Instruct and LLaVA-NeXT-7B show more than 10\% decrease in Attack Success Rate (ASR) together with an increment of at least 8\% in harmless dimension and over 13\% in helpful dimension of MLLMs on multi-modal multi-turn safety benchmarks, while preserving their general abilities.

AM$^3$Safety: Towards Data Efficient Alignment of Multi-modal Multi-turn Safety for MLLMs

TL;DR

AMSafety advances safety alignment for multi-turn multimodal LLMs by introducing InterSafe-V, a large-scale dialogue dataset with specialized refusal VQA samples, and a data-efficient training framework that couples a cold-start refusal phase with GRPO-based fine-tuning using a turn-aware dual-objective reward. The approach uses safety variance-based turn weighting to identify critical turns and balances helpfulness with safety via a dual reward, achieving over 10% ASR reduction and at least 8% improvement in harmlessness and over 13% in helpfulness on multi-turn benchmarks, while preserving general capabilities. The work emphasizes model-to-model data generation to reduce manual annotation costs and demonstrates strong performance on Qwen2.5-VL-7B-Instruct and LLaVA-NeXT-7B across SafeMT, JailbreakV, and MMSafe-PO, offering a practical, scalable path for safety in real-world multimodal dialogues.

Abstract

Multi-modal Large Language Models (MLLMs) are increasingly deployed in interactive applications. However, their safety vulnerabilities become pronounced in multi-turn multi-modal scenarios, where harmful intent can be gradually reconstructed across turns, and security protocols fade into oblivion as the conversation progresses. Existing Reinforcement Learning from Human Feedback (RLHF) alignment methods are largely developed for single-turn visual question-answer (VQA) task and often require costly manual preference annotations, limiting their effectiveness and scalability in dialogues. To address this challenge, we present InterSafe-V, an open-source multi-modal dialogue dataset containing 11,270 dialogues and 500 specially designed refusal VQA samples. This dataset, constructed through interaction between several models, is designed to more accurately reflect real-world scenarios and includes specialized VQA pairs tailored for specific domains. Building on this dataset, we propose AMSafety, a framework that combines a cold-start refusal phase with Group Relative Policy Optimization (GRPO) fine-tuning using turn-aware dual-objective rewards across entire dialogues. Experiments on Qwen2.5-VL-7B-Instruct and LLaVA-NeXT-7B show more than 10\% decrease in Attack Success Rate (ASR) together with an increment of at least 8\% in harmless dimension and over 13\% in helpful dimension of MLLMs on multi-modal multi-turn safety benchmarks, while preserving their general abilities.
Paper Structure (34 sections, 10 equations, 4 figures, 4 tables)

This paper contains 34 sections, 10 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dialogue Safety
  4. MLLMs Safety Alignment
  5. Methodology
  6. InterSafe-V
  7. Refusal VQA Pairs
  8. Step 1: Red-Team Queries Decomposition
  9. Step 2: Dialogue Construction
  10. Step 3: Data Cleaning
  11. AM$^3$Safety
  12. Refusal Template Learning
  13. Helpfulness Optimization
  14. Multi-Turn Dual-Objective Reward Function
  15. Experiments
  16. ...and 19 more sections

Figures (4)

  • Figure 1: We propose a three-step data construction pipeline. red-team queries sourced from existing multi-modal datasets are decomposed into multiple queries that are independently less harmful, while the combination of these queries retains the potential risk. Each query undergoes evaluation, with only those exceeding the threshold $\delta$ advancing to the subsequent step. These queries serve as prompts for simulated users during interactions with MLLMs and Qwen-image generates supplementary images based on descriptions as needed concurrently.
  • Figure 2: AM$^3$Safety Training pipeline: Cold-start establishes refusal capabilities in the policy model, followed by GRPO-based training that optimizes for helpfulness while maintaining safety alignment.
  • Figure 3: Comparison between cold-start only, GRPO-based fine-tuning only and AM$^3$Safety
  • Figure 4: