Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Metis-HOME: Hybrid Optimized Mixture-of-Experts for Multimodal Reasoning

Xiaohan Lan, Fanfan Liu, Haibo Qiu, Siqi Yang, Delian Ruan, Peng Shi, Lin Ma

TL;DR

Metis-HOME tackles the reasoning-vs-generalization dilemma in multimodal LLMs by introducing a hybrid optimized mixture-of-experts with two specialized branches: a thinking expert for multi-step reasoning and a non-thinking expert for rapid inference. A lightweight trainable router dynamically routes queries based on content and complexity, enabling adaptive computation. The model is built by expanding Qwen2.5-VL-7B into an MoE where FFNs are duplicated per block while self-attention is shared, and training proceeds in two stages: Stage-RL to enhance reasoning, followed by Stage-SFT with mixed thinking and non-thinking data. Results show substantial gains in reasoning benchmarks (avg ~46.1%, +6.9 points over Baseline) and a near-1 point improvement on eight general benchmarks (avg ~71.2%), indicating that the hybrid approach improves reasoning without sacrificing generalist capabilities. This work establishes a practical, scalable paradigm for versatile multimodal models and points to future expansion with more specialized experts and modalities.

Abstract

Inspired by recent advancements in LLM reasoning, the field of multimodal reasoning has seen remarkable progress, achieving significant performance gains on intricate tasks such as mathematical problem-solving. Despite this progress, current multimodal large reasoning models exhibit two key limitations. They tend to employ computationally expensive reasoning even for simple queries, leading to inefficiency. Furthermore, this focus on specialized reasoning often impairs their broader, more general understanding capabilities. In this paper, we propose Metis-HOME: a Hybrid Optimized Mixture-of-Experts framework designed to address this trade-off. Metis-HOME enables a ''Hybrid Thinking'' paradigm by structuring the original dense model into two distinct expert branches: a thinking branch tailored for complex, multi-step reasoning, and a non-thinking branch optimized for rapid, direct inference on tasks like general VQA and OCR. A lightweight, trainable router dynamically allocates queries to the most suitable expert. We instantiate Metis-HOME by adapting the Qwen2.5-VL-7B into an MoE architecture. Comprehensive evaluations reveal that our approach not only substantially enhances complex reasoning abilities but also improves the model's general capabilities, reversing the degradation trend observed in other reasoning-specialized models. Our work establishes a new paradigm for building powerful and versatile MLLMs, effectively resolving the prevalent reasoning-vs-generalization dilemma. Code and weights are available at https://github.com/MM-Thinking/Metis-HOME.

Metis-HOME: Hybrid Optimized Mixture-of-Experts for Multimodal Reasoning

TL;DR

Metis-HOME tackles the reasoning-vs-generalization dilemma in multimodal LLMs by introducing a hybrid optimized mixture-of-experts with two specialized branches: a thinking expert for multi-step reasoning and a non-thinking expert for rapid inference. A lightweight trainable router dynamically routes queries based on content and complexity, enabling adaptive computation. The model is built by expanding Qwen2.5-VL-7B into an MoE where FFNs are duplicated per block while self-attention is shared, and training proceeds in two stages: Stage-RL to enhance reasoning, followed by Stage-SFT with mixed thinking and non-thinking data. Results show substantial gains in reasoning benchmarks (avg ~46.1%, +6.9 points over Baseline) and a near-1 point improvement on eight general benchmarks (avg ~71.2%), indicating that the hybrid approach improves reasoning without sacrificing generalist capabilities. This work establishes a practical, scalable paradigm for versatile multimodal models and points to future expansion with more specialized experts and modalities.

Abstract

Inspired by recent advancements in LLM reasoning, the field of multimodal reasoning has seen remarkable progress, achieving significant performance gains on intricate tasks such as mathematical problem-solving. Despite this progress, current multimodal large reasoning models exhibit two key limitations. They tend to employ computationally expensive reasoning even for simple queries, leading to inefficiency. Furthermore, this focus on specialized reasoning often impairs their broader, more general understanding capabilities. In this paper, we propose Metis-HOME: a Hybrid Optimized Mixture-of-Experts framework designed to address this trade-off. Metis-HOME enables a ''Hybrid Thinking'' paradigm by structuring the original dense model into two distinct expert branches: a thinking branch tailored for complex, multi-step reasoning, and a non-thinking branch optimized for rapid, direct inference on tasks like general VQA and OCR. A lightweight, trainable router dynamically allocates queries to the most suitable expert. We instantiate Metis-HOME by adapting the Qwen2.5-VL-7B into an MoE architecture. Comprehensive evaluations reveal that our approach not only substantially enhances complex reasoning abilities but also improves the model's general capabilities, reversing the degradation trend observed in other reasoning-specialized models. Our work establishes a new paradigm for building powerful and versatile MLLMs, effectively resolving the prevalent reasoning-vs-generalization dilemma. Code and weights are available at https://github.com/MM-Thinking/Metis-HOME.
Paper Structure (18 sections, 4 equations, 7 figures, 3 tables)

This paper contains 18 sections, 4 equations, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Architectural Expansion
  4. Training Strategy
  5. Stage-RL
  6. Stage-SFT
  7. Experiment
  8. Implementation and Evaluation
  9. Quantitative Results
  10. Comparisons Against SOTA
  11. Thinking Ratio Analysis
  12. Qualitative Results
  13. Related Work
  14. Multimodal Reasoning.
  15. Hybrid Thinking Paradigm.
  16. ...and 3 more sections

Figures (7)

  • Figure 1: A brief illustration of hybrid thinking paradigm from Meits-HOME.
  • Figure 2: Performance comparisons across different benchmarks of Metis-HOME against baselines.
  • Figure 3: Thinking ratio of Metis-HOME across reasoning and general benchmarks.
  • Figure 4: Metis-HOME learns to assign more questions to its thinking expert, leading to progressively higher accuracy on MathVerse.
  • Figure 5: Trend of thinking ratio made by Metis-HOME during SFT training across different benchmarks.
  • ...and 2 more figures