Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contexts

TL;DR

The paper tackles the high computational cost of long-context modeling in transformer-based LLMs by introducing HALO, a data-efficient distillation pipeline that converts pre-trained Transformers into hybrid attention-RNN architectures. Building on HALO, it proposes HypeNet, a hybrid model enhanced by HyPE (Hybrid Positional Encoding) and other architectural tweaks to achieve superior length generalization and efficiency. Key results show HALO enables conversion with fewer than 3B tokens (about 0.01% of the teacher's data) and HyPE-based hybrids outperform state-of-the-art hybrids on long-context tasks while maintaining competitive CSR performance. These findings offer a practical path to cost-effective long-context LLMs suitable for applications requiring extended reasoning and memory, while enabling broader experimentation in hybrid architectures. Limitations include alignment of post-conversion capabilities and applicability to non-Transformer architectures.

Abstract

Hybrid Transformer architectures, which combine softmax attention blocks and recurrent neural networks (RNNs), have shown a desirable performance-throughput tradeoff for long-context modeling, but their adoption and studies are hindered by the prohibitive cost of large-scale pre-training from scratch. Some recent studies have shown that pre-trained softmax attention blocks can be converted into RNN blocks through parameter transfer and knowledge distillation. However, these transfer methods require substantial amounts of training data (more than 10B tokens), and the resulting hybrid models also exhibit poor long-context performance, which is the scenario where hybrid models enjoy significant inference speedups over Transformer-based models. In this paper, we present HALO (Hybrid Attention via Layer Optimization), a pipeline for distilling Transformer models into RNN-attention hybrid models. We then present HypeNet, a hybrid architecture with superior length generalization enabled by a novel position encoding scheme (named HyPE) and various architectural modifications. We convert the Qwen3 series into HypeNet using HALO, achieving performance comparable to the original Transformer models while enjoying superior long-context performance and efficiency. The conversion requires just 2.3B tokens, less than 0.01% of their pre-training data

Figures (8)

• Figure 1: Left & center: the performance-efficiency tradeoff of our model, HypeNet, versus the Qwen3 series, measured with 128K context length and BFloat16 precision. Right: the time per output token of the 1.7B models across different context lengths. For 1M context length, the Qwen3 model runs out of GPU memory. HypeNet is converted from Qwen3 using our distillation procedure, HALO, and has better performance-efficiency tradeoff than Qwen3.
• Figure 2: Various pipelines for converting Transformer models into hybrid models. The boxes with dotted lines represent training-free stages, while those with solid lines represent training stages. HALO is much more data-efficient than prior methods.
• Figure 3: Illustration of HypeNet. The architectural modifications introduced during HALO are marked with ➊, ➋, ➌, and ➍. Red dotted lines indicate components that are removed during HALO, black dotted lines indicate components that are added.
• Figure 4: NIAH scores of HypeNet variants based on different position encodings, as a function of context length. The models are trained from scratch with 20B tokens and 500M parameters.
• Figure 5: NIAH scores of HypeNet variants based on different RNN mixers, as a function of context length. The models are trained from scratch with 20B tokens and 500M parameters.