Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Context Scaling with LongCat ZigZag Attention

Chen Zhang, Yang Bai, Jiahuan Li, Anchun Gui, Keheng Wang, Feifan Liu, Guanyu Wu, Yuwei Jiang, Defei Bu, Li Wei, Haihang Jing, Hongyin Tang, Xin Chen, Xiangzhou Huang, Fengcun Li, Rongxiang Weng, Yulei Qian, Yifan Lu, Yerui Sun, Jingang Wang, Yuchen Xie, Xunliang Cai

TL;DR

This work tackles the quadratic context-cost of full-attention by introducing LongCat ZigZag Attention (LoZA), a sparsification pipeline that calibrates and sparsifiesMLAs during mid-training and then retrains to recover performance. By employing layer-level sparsity and a calibrated mix of full- and sparse-attention paths, LoZA enables long-context processing, demonstrated on LongCat-Flash with LongCat-Flash-Exp extending to up to 1M tokens via YaRN extrapolation. Evaluations show that LoZA preserves effectiveness across a wide range of benchmarks while delivering significant efficiency gains in prefill and decode phases, making context-native reasoning and long-horizon agentic tasks more viable. The approach offers a general pathway to retrofit existing full-attention models into sparse architectures without prohibitive compute, with potential broad impact on sparse-attention research and open-source multimodal models.

Abstract

We introduce LongCat ZigZag Attention (LoZA), which is a sparse attention scheme designed to transform any existing full-attention models into sparse versions with rather limited compute budget. In long-context scenarios, LoZA can achieve significant speed-ups both for prefill-intensive (e.g., retrieval-augmented generation) and decode-intensive (e.g., tool-integrated reasoning) cases. Specifically, by applying LoZA to LongCat-Flash during mid-training, we serve LongCat-Flash-Exp as a long-context foundation model that can swiftly process up to 1 million tokens, enabling efficient long-term reasoning and long-horizon agentic capabilities.

Efficient Context Scaling with LongCat ZigZag Attention

TL;DR

This work tackles the quadratic context-cost of full-attention by introducing LongCat ZigZag Attention (LoZA), a sparsification pipeline that calibrates and sparsifiesMLAs during mid-training and then retrains to recover performance. By employing layer-level sparsity and a calibrated mix of full- and sparse-attention paths, LoZA enables long-context processing, demonstrated on LongCat-Flash with LongCat-Flash-Exp extending to up to 1M tokens via YaRN extrapolation. Evaluations show that LoZA preserves effectiveness across a wide range of benchmarks while delivering significant efficiency gains in prefill and decode phases, making context-native reasoning and long-horizon agentic tasks more viable. The approach offers a general pathway to retrofit existing full-attention models into sparse architectures without prohibitive compute, with potential broad impact on sparse-attention research and open-source multimodal models.

Abstract

We introduce LongCat ZigZag Attention (LoZA), which is a sparse attention scheme designed to transform any existing full-attention models into sparse versions with rather limited compute budget. In long-context scenarios, LoZA can achieve significant speed-ups both for prefill-intensive (e.g., retrieval-augmented generation) and decode-intensive (e.g., tool-integrated reasoning) cases. Specifically, by applying LoZA to LongCat-Flash during mid-training, we serve LongCat-Flash-Exp as a long-context foundation model that can swiftly process up to 1 million tokens, enabling efficient long-term reasoning and long-horizon agentic capabilities.
Paper Structure (11 sections, 4 equations, 3 figures, 3 tables)

This paper contains 11 sections, 4 equations, 3 figures, 3 tables.

Table of Contents

  1. Architecture
  2. Full Attention
  3. LongCat ZigZag Attention
  4. Desiderata
  5. Training
  6. Mid-training
  7. Post-training
  8. Evaluation
  9. Effectiveness
  10. Efficiency
  11. Conclusion

Figures (3)

  • Figure 1: The illustration of LongCat ZigZag Attention (LoZA), which involves first calibration and then training for realizing the sparsity. The illustration is shown with the exemplar shortcuted-MoE DBLP:conf/icml/CaiJQC0025 in LongCat-Flash, which encompasses two MLA layers DBLP:journals/corr/abs-2405-04434. SSA: streaming sparse attention DBLP:conf/iclr/XiaoTCHL24.
  • Figure 2: The effectiveness of LongCat-Flash-Exp-Chat across different context lengths on MRCR. Qwen-3 is considered as a competitive baseline since it as well possesses the ability of handling 1M context. AUC: area under curve.
  • Figure 3: The efficiency of LoZA. The relative cost and speed-up are practically measured on H20 clusters.