MrRoPE: Mixed-radix Rotary Position Embedding

TL;DR

This paper reframes RoPE context-extension as mixed-radix radix conversion, unifying prior approaches under the MrRoPE framework. It introduces two training-free methods, MrRoPE-Uni and MrRoPE-Pro, with the latter using progressive radix expansion to preserve high-frequency information while extending the context, yielding superior perplexity and retrieval performance (e.g., extending effective context to 28K+ in theory and nearly 96K in practice). Theoretical analysis via RoPE Bound Theory and attention-score diagnostics substantiates the larger encoding length and stabilized intermediate-dimension dynamics. Empirically, MrRoPE-Pro outperforms YaRN and NTK baselines on long-context benchmarks like Needle-in-a-Haystack and Infinite-Bench, approaching fine-tuned long-context models without additional training. Overall, the work provides a principled, scalable path to training-free long-context inference for RoPE-based LLMs with strong practical impact.

Abstract

Rotary Position Embedding (RoPE)-extension refers to modifying or generalizing the Rotary Position Embedding scheme to handle longer sequences than those encountered during pre-training. However, current extension strategies are highly diverse and lack a unified theoretical foundation. In this paper, we propose MrRoPE (Mixed-radix RoPE), a generalized encoding formulation based on a radix system conversion perspective, which elegantly unifies various RoPE-extension approaches as distinct radix conversion strategies. Based on this theory, we introduce two training-free extensions, MrRoPE-Uni and MrRoPE-Pro, which leverage uniform and progressive radix conversion strategies, respectively, to achieve 'train short, test long' generalization. Without fine-tuning, MrRoPE-Pro sustains over 85% recall in the 128K-context Needle-in-a-Haystack test and achieves more than double YaRN's accuracy on Infinite-Bench retrieval and dialogue subsets. Theoretical analysis confirms that MrRoPE-Pro effectively raises the upper bound of RoPE's attainable encoding length, which further validates the reliability and utility of our theory and methodology.

Figures (8)

• Figure 1: The overall framework of our work. Our key contributions are: (1) a unified theoretical framework for major RoPE-extensions, reflecting them into a specific radix conversion behavior; (2) a progressive radix conversion method MrRoPE-Pro, which outperforms other SoTA methods across various tasks.
• Figure 2: The biased positional estimate $\hat{m}$ of RoPE across different base.
• Figure 3: The cumulative scaling factor $s_d$ of different RoPE extension methods across varying dimension index.(Scale-up to 16x and 4x)
• Figure 4: In the Needle-IN-A-Haystack test, MrRoPE-Pro (right) effectively extends LLaMA3-8B's context window to nearly 96K, which is much longer than the performance of YaRN (left).
• Figure 5: The cosine sum of the rotation angles in each dimension, measuring the ability to give more attention to similar tokens than a random one. The base value and original context length are consistent with the settings of LLaMA2-7B.