Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoning

TL;DR

The work tackles the challenge of long-context reasoning in large language models by proposing Ring-linear, a hybrid architecture that blends linear attention with softmax attention within a modular, MoE-enabled framework. It introduces two open-sourced models, Ring-mini-linear-2.0 and Ring-flash-linear-2.0, and demonstrates large reductions in inference and training costs ($1/10$ and $>50\%$ respectively) while maintaining state-of-the-art performance on complex reasoning benchmarks. Key contributions include a detailed design of hybrid attention, kernel-level FP8 optimizations via LingHe, and a systematic RL alignment strategy that stabilizes long-horizon training, enabling efficient continual pre-training, post-training, and RL. The approach offers practical benefits for real-world long-context applications by delivering substantial speedups in both prefill and decoding phases and by improving training stability under reinforcement learning, albeit with memory overhead and remaining bottlenecks in softmax segments that the authors plan to address in future work.

Abstract

In this technical report, we present the Ring-linear model series, specifically including Ring-mini-linear-2.0 and Ring-flash-linear-2.0. Ring-mini-linear-2.0 comprises 16B parameters and 957M activations, while Ring-flash-linear-2.0 contains 104B parameters and 6.1B activations. Both models adopt a hybrid architecture that effectively integrates linear attention and softmax attention, significantly reducing I/O and computational overhead in long-context inference scenarios. Compared to a 32 billion parameter dense model, this series reduces inference cost to 1/10, and compared to the original Ring series, the cost is also reduced by over 50%. Furthermore, through systematic exploration of the ratio between different attention mechanisms in the hybrid architecture, we have identified the currently optimal model structure. Additionally, by leveraging our self-developed high-performance FP8 operator library-linghe, overall training efficiency has been improved by 50%. Benefiting from the high alignment between the training and inference engine operators, the models can undergo long-term, stable, and highly efficient optimization during the reinforcement learning phase, consistently maintaining SOTA performance across multiple challenging complex reasoning benchmarks.

Figures (13)

• Figure 1: Benchmark results of Ring-flash-linear-2.0 versus counterparts on representative metrics.
• Figure 2: Model architecture of Ring-linear
• Figure 3: Scaling law curves. (Left) Curves fitted with different runs of the hybrid linear architecture ($M=1$) and the softmax attention architecture. (Right) Curves comparing different layer group size ($M$) configurations.
• Figure 4: Variation of KV cache/State memory access size with increasing sequence length.
• Figure 5: Overview of computation optimization