Metis-SPECS: Decoupling Multimodal Learning via Self-distilled Preference-based Cold Start

TL;DR

Metis-SPECS tackles the cold-start bottleneck in multimodal RL by decoupling shallow, format-focused learning from deep reasoning. It introduces a three-stage SPECS pipeline that uses self-distilled preference data and Direct Preference Optimization (DPO) to pre-align a model before final GRPO fine-tuning with verifiable rewards. The Generalization Factor (GF) provides a metric to assess how well a cold-start method generalizes to in-distribution and out-of-distribution tasks, guiding the design of more robust starting points. Across benchmarks such as MEGA-Bench and MathVista, SPECS yields consistent gains and improved training stability, illustrating the value of separating cold-start objectives from RL optimization and of data-generation strategies aligned with the model’s distribution.

Abstract

Reinforcement learning (RL) with verifiable rewards has recently catalyzed a wave of "MLLM-r1" approaches that bring RL to vision language models. Most representative paradigms begin with a cold start, typically employing supervised fine-tuning (SFT), to initialize the policy before RL. However, SFT-based cold start adopts the reasoning paradigm intertwined with task solution and output format, which may induce instruction-style overfitting, weakens out-of-distribution generalization, and ultimately affects downstream RL. We revisit the cold start along two views, its training method and data construction, and introduce the Generalization Factor (GF) coefficient to quantify the generalization capability under different methods. Our empirical study finds that preference-based training methods (e.g. DPO) generalizes better than SFT-based methods in cold start. Motivated by this, we propose SPECS-a Self-distilled, Preference-based Cold Start framework that decouples multimodal learning: (1) generates introspective preference data pairs via self-distillation, avoiding reliance on larger teachers or manual annotation; (2) performs preference-based training to learn, focusing on shallow, transferable surface-form criteria (format, structure, style) rather than memorizing content; and (3) hands off to RL with verifiable rewards for deep reasoning results. Experimental results across multiple multimodal benchmarks show that our decoupling learning framework yields consistent performance gains over strong baselines, improving MEGA-Bench by 4.1% and MathVista by 12.2%. Additional experiments indicate that SPECS contributes to reducing in-distribution "stuckness," improving exploration, stabilizing training, and raising the performance ceiling.

Figures (7)

• Figure 1: Performance Comparison: DPO vs. SFT on In-Distribution and Out-of-Distribution Task
• Figure 2: Method Overview. We propose the SPECS cold-start strategy, a three-stage pipeline to enhance final RL fine-tuning. Firstly, where we generate a preference dataset focused on teaching the correct output format by self distillation. Next, The base model is pre-aligned on this data using DPO to create a format-aware "Warmup Model". Finally, this pre-aligned model undergoes Final RL tuning with GRPO, allowing the optimization process to focus on enhancing reasoning.
• Figure 3: Example of a self-distilled preference data pair.
• Figure 4: Impact on RL Training Efficiency and stability.
• Figure 5: GF vs. Final Performance