Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

COSMO-RL: Towards Trustworthy LMRMs via Joint Safety and Stability

Yizhuo Ding, Mingkang Chen, Qiuhua Liu, Fenghua Weng, Wanying Qu, Yue Yang, Yugang Jiang, Zuxuan Wu, Yanwei Fu, Wenqi Shao

TL;DR

This work tackles the safety-capability trade-off in large multimodal reasoning models by introducing CoSMo-RL, a unified reinforcement learning framework that co-optimizes safety, value, and general reasoning in a two-stage schedule. It couples supervised fine-tuning with a Clipped Policy Gradient with Policy Drift (CPGD) objective and a four-component multiobjective reward, while augmenting training with multimodal jailbreak data and dedicated reward models (Safety, Value, Knowledge). Empirical results show consistent gains in safety, value alignment, and multimodal reasoning across multiple backbones and benchmarks, with robust resistance to jailbreaks and reduced unnecessary refusals. The approach demonstrates that safety and capability can co-evolve in a stable pipeline, providing transferable improvements and practical guidance for deploying trustworthy LMRMs in real-world settings.

Abstract

Large Multimodal Reasoning Models (LMRMs) are moving into real applications, where they must be both useful and safe. Safety is especially challenging in multimodal settings: images and text can be combined to bypass guardrails, and single objective training can cause policy drift that yields over-refusal on benign inputs or unsafe compliance on risky ones. We present COSMO-RL, a mixed reinforcement learning framework that trains reasoning oriented LMRMs under multimodal, multitask, and multiobjective signals, and we release the resulting model, COSMO-R1. Our approach aims to let safety and capability grow together in one stable pipeline rather than competing during alignment. In experiments, COSMO-R1 improves safety while maintaining-and often improving multimodal reasoning and instruction following, shows stronger robustness to multimodal jailbreaks, and reduces unnecessary refusals. The framework also transfers across backbones with consistent gains. Ablations support the design choices, indicating a simple path to advancing safety and general capability together in LMRMs.

COSMO-RL: Towards Trustworthy LMRMs via Joint Safety and Stability

TL;DR

This work tackles the safety-capability trade-off in large multimodal reasoning models by introducing CoSMo-RL, a unified reinforcement learning framework that co-optimizes safety, value, and general reasoning in a two-stage schedule. It couples supervised fine-tuning with a Clipped Policy Gradient with Policy Drift (CPGD) objective and a four-component multiobjective reward, while augmenting training with multimodal jailbreak data and dedicated reward models (Safety, Value, Knowledge). Empirical results show consistent gains in safety, value alignment, and multimodal reasoning across multiple backbones and benchmarks, with robust resistance to jailbreaks and reduced unnecessary refusals. The approach demonstrates that safety and capability can co-evolve in a stable pipeline, providing transferable improvements and practical guidance for deploying trustworthy LMRMs in real-world settings.

Abstract

Large Multimodal Reasoning Models (LMRMs) are moving into real applications, where they must be both useful and safe. Safety is especially challenging in multimodal settings: images and text can be combined to bypass guardrails, and single objective training can cause policy drift that yields over-refusal on benign inputs or unsafe compliance on risky ones. We present COSMO-RL, a mixed reinforcement learning framework that trains reasoning oriented LMRMs under multimodal, multitask, and multiobjective signals, and we release the resulting model, COSMO-R1. Our approach aims to let safety and capability grow together in one stable pipeline rather than competing during alignment. In experiments, COSMO-R1 improves safety while maintaining-and often improving multimodal reasoning and instruction following, shows stronger robustness to multimodal jailbreaks, and reduces unnecessary refusals. The framework also transfers across backbones with consistent gains. Ablations support the design choices, indicating a simple path to advancing safety and general capability together in LMRMs.

Paper Structure

This paper contains 29 sections, 4 equations, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Safety Alignment for Vision--Language Models
  4. Mixed RL Training
  5. Method
  6. Supervised Fine-Tuning
  7. The CPGD Algorithm
  8. Multitask Training Pipeline
  9. Multiobjective Reward Function
  10. Multimodal Jailbreak Data Augmentation
  11. Experiments
  12. Data
  13. Evaluation of CoSMo-R1
  14. Safety Evaluation
  15. Value Evaluation
  16. ...and 14 more sections

Figures (4)

  • Figure 1: Overview of the CoSMo-RL framework. After SFT, RL Training proceeds in two stages: Stage 1 augments the model's General capability; Stage 2 jointly optimizes Safety, Value, and General. During RL, each capability track is guided by a multiobjective reward composed of Format, Visual-Focus, Helpful, and Task-Aware terms. The framework is explicitly multimodal, multitask, and multiobjective, covering both visual and text inputs.
  • Figure 2: An attack example in MM-SafetyBench by using text, images, and image-guided words.
  • Figure 3: CoSMo-RL data augmentation for text and vision.
  • Figure 4: More Cases on MM-SafetyBench and SIUO BenchMark.