Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generation

Inclusion AI, :, Bowen Ma, Cheng Zou, Canxiang Yan, Chunxiang Jin, Chunjie Shen, Chenyu Lian, Dandan Zheng, Fudong Wang, Furong Xu, GuangMing Yao, Jun Zhou, Jingdong Chen, Jianing Li, Jianxin Sun, Jiajia Liu, Jian Sha, Jianjiang Zhu, Jianping Jiang, Jun Peng, Kaixiang Ji, Kaimeng Ren, Libin Wang, Lixiang Ru, Longhua Tan, Lu Ma, Lan Wang, Mochen Bai, Ning Gao, Qingpei Guo, Qinglong Zhang, Qiang Xu, Rui Liu, Ruijie Xiong, Ruobing Zheng, Sirui Gao, Tao Zhang, Tianqi Li, Tinghao Liu, Weilong Chai, Xinyu Xiao, Xiaomei Wang, Xiaolong Wang, Xiao Lu, Xiaoyu Li, Xingning Dong, Xuzheng Yu, Yi Yuan, Yuting Gao, Yuting Xiao, Yunxiao Sun, Yipeng Chen, Yifan Mao, Yifei Wu, Yongjie Lyu, Ziping Ma, Zhiqiang Fang, Zhihao Qiu, Ziyuan Huang, Zizheng Yang, Zhengyu He

TL;DR

Ming-Flash-Omni tackles unified multimodal perception and generation across vision, speech, and language by adopting a sparse Mixture-of-Experts backbone (Ling-Flash-2.0) with 100B total parameters and 6.1B active per token, enabling large capacity with efficient inference. It integrates VideoRoPE for temporality, continuous acoustic latents for speech generation, and generative segmentation for tight coupling of understanding and editing, achieving state-of-the-art results across ContextASR, text-to-image generation, and generative segmentation. The work introduces a two-stage training pipeline with advanced RL (D-GRPO, U-DPO) and a diffusion-based image generator, supported by data-heterogeneity-aware infrastructure (sequence packing, encoder sharding) and a diverse, multi-modal dataset. Overall, Ming-Flash-Omni demonstrates far-reaching multimodal capabilities with improved efficiency, broad task coverage, and open-source release aimed at progressing toward Artificial General Intelligence.

Abstract

We propose Ming-Flash-Omni, an upgraded version of Ming-Omni, built upon a sparser Mixture-of-Experts (MoE) variant of Ling-Flash-2.0 with 100 billion total parameters, of which only 6.1 billion are active per token. This architecture enables highly efficient scaling (dramatically improving computational efficiency while significantly expanding model capacity) and empowers stronger unified multimodal intelligence across vision, speech, and language, representing a key step toward Artificial General Intelligence (AGI). Compared to its predecessor, the upgraded version exhibits substantial improvements across multimodal understanding and generation. We significantly advance speech recognition capabilities, achieving state-of-the-art performance in contextual ASR and highly competitive results in dialect-aware ASR. In image generation, Ming-Flash-Omni introduces high-fidelity text rendering and demonstrates marked gains in scene consistency and identity preservation during image editing. Furthermore, Ming-Flash-Omni introduces generative segmentation, a capability that not only achieves strong standalone segmentation performance but also enhances spatial control in image generation and improves editing consistency. Notably, Ming-Flash-Omni achieves state-of-the-art results in text-to-image generation and generative segmentation, and sets new records on all 12 contextual ASR benchmarks, all within a single unified architecture.

Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generation

TL;DR

Ming-Flash-Omni tackles unified multimodal perception and generation across vision, speech, and language by adopting a sparse Mixture-of-Experts backbone (Ling-Flash-2.0) with 100B total parameters and 6.1B active per token, enabling large capacity with efficient inference. It integrates VideoRoPE for temporality, continuous acoustic latents for speech generation, and generative segmentation for tight coupling of understanding and editing, achieving state-of-the-art results across ContextASR, text-to-image generation, and generative segmentation. The work introduces a two-stage training pipeline with advanced RL (D-GRPO, U-DPO) and a diffusion-based image generator, supported by data-heterogeneity-aware infrastructure (sequence packing, encoder sharding) and a diverse, multi-modal dataset. Overall, Ming-Flash-Omni demonstrates far-reaching multimodal capabilities with improved efficiency, broad task coverage, and open-source release aimed at progressing toward Artificial General Intelligence.

Abstract

We propose Ming-Flash-Omni, an upgraded version of Ming-Omni, built upon a sparser Mixture-of-Experts (MoE) variant of Ling-Flash-2.0 with 100 billion total parameters, of which only 6.1 billion are active per token. This architecture enables highly efficient scaling (dramatically improving computational efficiency while significantly expanding model capacity) and empowers stronger unified multimodal intelligence across vision, speech, and language, representing a key step toward Artificial General Intelligence (AGI). Compared to its predecessor, the upgraded version exhibits substantial improvements across multimodal understanding and generation. We significantly advance speech recognition capabilities, achieving state-of-the-art performance in contextual ASR and highly competitive results in dialect-aware ASR. In image generation, Ming-Flash-Omni introduces high-fidelity text rendering and demonstrates marked gains in scene consistency and identity preservation during image editing. Furthermore, Ming-Flash-Omni introduces generative segmentation, a capability that not only achieves strong standalone segmentation performance but also enhances spatial control in image generation and improves editing consistency. Notably, Ming-Flash-Omni achieves state-of-the-art results in text-to-image generation and generative segmentation, and sets new records on all 12 contextual ASR benchmarks, all within a single unified architecture.

Paper Structure

This paper contains 25 sections, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Ming-Flash-Omni
  3. Unified Understanding Across Modalities
  4. Unified Speech Understanding and Generation
  5. Unified Image Understanding and Generation
  6. Generative Segmentation as an Editing Task
  7. Empowering Advanced Controllable Capabilities
  8. Overall Training Procedure
  9. Infrastructure
  10. Data Construction
  11. Image Data
  12. Image Understanding Data
  13. Image Generation Data
  14. Audio Data
  15. Video Data
  16. ...and 10 more sections

Figures (5)

  • Figure 1: Ming-Flash-Omni generally demonstrates highly competitive performance across various domains, including vision-text understanding, controllable image generation, speech recognition, and speech synthesis. Specifically, in image generation, Ming-Flash-Omni introduces a novel generative segmentation paradigm to achieve fine-grained spatial and semantic control over the generated images. Moreover, Ming-Flash-Omni significantly enhances Context-Aware Speech Recognition (ContextASR) and Chinese dialect recognition, thereby broadening its applicability in real-world scenarios.
  • Figure 2: The overall framework of Ming-Flash-Omni. This version features a sparser LLM based on Ling-flash-2.0 MoE architecture, and integrates VideoRoPE to enhance temporal modeling. Speech generation now uses continuous features instead of discrete tokens, and image generation has been upgraded with support for segmentation.
  • Figure 3: Visualization results of Ming-Flash-Omni on understanding tasks, including world knowledge, multi-image understanding, mathematical reasoning, OCR, contextual ASR, and dialect-aware ASR.
  • Figure 4: Visualization results of Ming-Flash-Omni on Text/Image $\rightarrow$ Image tasks, including image generation task, image editing task, and image segmentation task.
  • Figure 5: Visualization results of Ming-Flash-Omni on Image $\rightarrow$ Image tasks, including ID photo generation, ID photo editing, background replacement, and multi-image editing.