Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LongCat-Flash-Omni Technical Report

Meituan LongCat Team, Bairui Wang, Bayan, Bin Xiao, Bo Zhang, Bolin Rong, Borun Chen, Chang Wan, Chao Zhang, Chen Huang, Chen Chen, Chen Chen, Chengxu Yang, Chengzuo Yang, Cong Han, Dandan Peng, Delian Ruan, Detai Xin, Disong Wang, Dongchao Yang, Fanfan Liu, Fengjiao Chen, Fengyu Yang, Gan Dong, Gang Huang, Gang Xu, Guanglu Wan, Guoqiang Tan, Guoqiao Yu, Haibo Qiu, Hao Lu, Hongbo Liu, Hongyu Xiang, Jiaheng Wu, Jian Yang, Jiaxing Liu, Jing Huang, Jingang Wang, Jinrui Ding, Juchao Jiang, Jun Kuang, Jun Wang, Junhui Mei, Ke Ding, Kefeng Zhang, Lei Chen, Liang Shi, Limeng Qiao, Liming Zheng, Lin Ma, Liuyang Guo, Liya Ma, Luying Sun, Man Gao, Mengshen Zhu, Miao Cao, Minliang Lin, Nuo Xu, Peng Shi, Qi Zhang, Qian Fang, Qian Wang, Qian Yang, Quanxiu Wang, Rongxiang Weng, Rongxin Guo, Ruoxuan Liang, Senbin Yang, Shanbo Xu, Shanglin Lei, Shengze Ye, Shimin Chen, Shuaiqi Chen, Shujie Hu, Shuo Li, Siqi Yang, Siyu Xu, Siyu Ren, Song Li, Songxiang Liu, Tianhao Bai, Tianye Dai, Wei Hong, Wei Wang, Weixiao Zhao, Wengang Cao, Wenlong Zhu, Wenlong He, Xi Su, Xi Nan, Xiaohan Zhao, Xiaohao Wang, Xiaoyu Zhao, Xiaoyu Wang, Xiaoyu Li, Xin Pan, Xin Chen, Xiusong Sun, Xu Xiang, Xudong Xing, Xuezhi Cao, Xunliang Cai, Yang Yang, Yanli Tan, Yao Yao, Yerui Sun, Yi Chen, Yifan Lu, Yin Gong, Yining Zhang, Yitian Chen, Yiyang Gan, Yuchen Tang, Yuchen Xie, Yueqian Wang, Yuewen Zheng, Yufei Zhang, Yufeng Zhong, Yulei Qian, Yuqi Peng, Yuqian Li, Yuwei Jiang, Zeyang Hu, Zheng Zhang, Zhengkun Tian, Zhiqing Hong, Zhixiong Zeng, Zhuqi Mi, Ziran Li, Ziwen Wang, Ziyi Zhao, Ziyuan Zhuang, Zizhe Zhao

TL;DR

LongCat-Flash-Omni introduces a 560B open-source omni-modal model that unifies text, image, video, and audio with real-time streaming. The approach combines a curriculum-driven, multi-stage pretraining pipeline with modality-decoupled parallelism to scale training efficiency and maintain high unimodal performance. Key innovations include a lightweight vision/audio stack, a ScMoE backbone, dynamic video sampling and streaming interleaving, and a decoupled inference framework, enabling millisecond-level interactions and 128K-token context reasoning. Extensive evaluations demonstrate state-of-the-art omni-modal benchmarks, strong audio/text/image/video capabilities, and competitive real-time interaction quality, underscoring its potential as a foundation for AGI-oriented human-AI interfaces.

Abstract

We introduce LongCat-Flash-Omni, a state-of-the-art open-source omni-modal model with 560 billion parameters, excelling at real-time audio-visual interaction. By adopting a curriculum-inspired progressive training strategy that transitions from simpler to increasingly complex modality sequence modeling tasks, LongCat-Flash-Omni attains comprehensive multimodal capabilities while maintaining strong unimodal capability. Building upon LongCat-Flash, which adopts a high-performance Shortcut-connected Mixture-of-Experts (MoE) architecture with zero-computation experts, LongCat-Flash-Omni integrates efficient multimodal perception and speech reconstruction modules. Despite its immense size of 560B parameters (with 27B activated), LongCat-Flash-Omni achieves low-latency real-time audio-visual interaction. For training infrastructure, we developed a modality-decoupled parallelism scheme specifically designed to manage the data and model heterogeneity inherent in large-scale multimodal training. This innovative approach demonstrates exceptional efficiency by sustaining over 90% of the throughput achieved by text-only training. Extensive evaluations show that LongCat-Flash-Omni achieves state-of-the-art performance on omni-modal benchmarks among open-source models. Furthermore, it delivers highly competitive results across a wide range of modality-specific tasks, including text, image, and video understanding, as well as audio understanding and generation. We provide a comprehensive overview of the model architecture design, training procedures, and data strategies, and open-source the model to foster future research and development in the community.

LongCat-Flash-Omni Technical Report

TL;DR

LongCat-Flash-Omni introduces a 560B open-source omni-modal model that unifies text, image, video, and audio with real-time streaming. The approach combines a curriculum-driven, multi-stage pretraining pipeline with modality-decoupled parallelism to scale training efficiency and maintain high unimodal performance. Key innovations include a lightweight vision/audio stack, a ScMoE backbone, dynamic video sampling and streaming interleaving, and a decoupled inference framework, enabling millisecond-level interactions and 128K-token context reasoning. Extensive evaluations demonstrate state-of-the-art omni-modal benchmarks, strong audio/text/image/video capabilities, and competitive real-time interaction quality, underscoring its potential as a foundation for AGI-oriented human-AI interfaces.

Abstract

We introduce LongCat-Flash-Omni, a state-of-the-art open-source omni-modal model with 560 billion parameters, excelling at real-time audio-visual interaction. By adopting a curriculum-inspired progressive training strategy that transitions from simpler to increasingly complex modality sequence modeling tasks, LongCat-Flash-Omni attains comprehensive multimodal capabilities while maintaining strong unimodal capability. Building upon LongCat-Flash, which adopts a high-performance Shortcut-connected Mixture-of-Experts (MoE) architecture with zero-computation experts, LongCat-Flash-Omni integrates efficient multimodal perception and speech reconstruction modules. Despite its immense size of 560B parameters (with 27B activated), LongCat-Flash-Omni achieves low-latency real-time audio-visual interaction. For training infrastructure, we developed a modality-decoupled parallelism scheme specifically designed to manage the data and model heterogeneity inherent in large-scale multimodal training. This innovative approach demonstrates exceptional efficiency by sustaining over 90% of the throughput achieved by text-only training. Extensive evaluations show that LongCat-Flash-Omni achieves state-of-the-art performance on omni-modal benchmarks among open-source models. Furthermore, it delivers highly competitive results across a wide range of modality-specific tasks, including text, image, and video understanding, as well as audio understanding and generation. We provide a comprehensive overview of the model architecture design, training procedures, and data strategies, and open-source the model to foster future research and development in the community.

Paper Structure

This paper contains 60 sections, 2 equations, 13 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Architecture
  3. Vision Encoder
  4. Audio Tokenizer, Encoder, and Decoder
  5. LLM Backbone
  6. Video Strategy and Streaming Audio-Visual Interaction
  7. Video Strategy
  8. Streaming Audio-Visual Interaction
  9. Pre-Training
  10. Data Curation
  11. Audio Data
  12. Generic Image-Text Data
  13. OCR, Grounding and GUI Data
  14. STEM Data
  15. Multi-Image Data
  16. ...and 45 more sections

Figures (13)

  • Figure 1: Benchmark performance of LongCat-Flash-Omni.
  • Figure 2: An overview of the LongCat-Flash-Omni model architecture. The model is fully end-to-end and unifies multimodal understanding and generation across text, image, video, and audio within a single large language model framework. An vision encoder and an audio encoder are used to obtain vision features and audio features, respectively, which are then projected into a shared latent token space and fed into the LongCat-Flash LLM backbone. The LLM decoder directly generates multi-codebook speech tokens, parallel to generated text tokens, which are then converted to audio waveforms by an audio decoder. Shortcut-connected MoE (ScMoE) with zero-computation experts module proposed in LongCat-Flash is employed to achieve efficient multimodal fusion. Vision and audio features are chunk-wisely interleaved to support streaming audio-visual input.
  • Figure 2: Computation distribution per micro-batch across different modalities during the SFT stage.
  • Figure 3: Architecture of the audio decoder.
  • Figure 4: Architecture of the audio encoder.
  • ...and 8 more figures