Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CRoPE: Efficient Parametrization of Rotary Positional Embedding

Beicheng Lou, Zifei Xu

TL;DR

CRoPE reframes rotary positional embedding by parameterizing the $Q$, $K$, and $V$ projections as complex linear transformations, yielding about a 50% reduction in attention-block parameters. The authors show that the reduced function space of CRoPE is redundant in simple analytic tasks and that, in practice, CRoPE delivers comparable performance to RoPE on standard datasets (WikiText-2, PTB, PG-19) with substantially fewer parameters. Using a small GPT-2–style model ($L=4$, $H=4$, $d_{ ext{model}}=128$), CRoPE achieves validation losses close to RoPE while reducing attention-related parameters, with larger savings when counting additional components. Overall, CRoPE offers a parameter-efficient, interpretable alternative to RoPE that maintains performance while simplifying the embedding space.

Abstract

Rotary positional embedding has become the state-of-the-art approach to encode position information in transformer-based models. While it is often succinctly expressed in complex linear algebra, we note that the actual implementation of $Q/K/V$-projections is not equivalent to a complex linear transformation. We argue that complex linear transformation is a more natural parametrization and saves near 50\% parameters within the attention block. We show empirically that removing such redundancy has negligible impact on the model performance both in sample and out of sample. Our modification achieves more efficient parameter usage, as well as a cleaner interpretation of the representation space.

CRoPE: Efficient Parametrization of Rotary Positional Embedding

TL;DR

CRoPE reframes rotary positional embedding by parameterizing the , , and projections as complex linear transformations, yielding about a 50% reduction in attention-block parameters. The authors show that the reduced function space of CRoPE is redundant in simple analytic tasks and that, in practice, CRoPE delivers comparable performance to RoPE on standard datasets (WikiText-2, PTB, PG-19) with substantially fewer parameters. Using a small GPT-2–style model (, , ), CRoPE achieves validation losses close to RoPE while reducing attention-related parameters, with larger savings when counting additional components. Overall, CRoPE offers a parameter-efficient, interpretable alternative to RoPE that maintains performance while simplifying the embedding space.

Abstract

Rotary positional embedding has become the state-of-the-art approach to encode position information in transformer-based models. While it is often succinctly expressed in complex linear algebra, we note that the actual implementation of -projections is not equivalent to a complex linear transformation. We argue that complex linear transformation is a more natural parametrization and saves near 50\% parameters within the attention block. We show empirically that removing such redundancy has negligible impact on the model performance both in sample and out of sample. Our modification achieves more efficient parameter usage, as well as a cleaner interpretation of the representation space.
Paper Structure (20 sections, 25 equations, 4 figures, 2 tables)

This paper contains 20 sections, 25 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background and related work
  3. Absolute and relative positional embedding
  4. Rotary Positional Embedding
  5. Related work
  6. Complex Rotary Positional Embedding (CRoPE)
  7. Origin
  8. Detailed look into function space
  9. Illustrative Example
  10. Simple token comparison
  11. Simple position comparison
  12. Token-dependent position comparison
  13. Experiments
  14. Model architecture
  15. Datasets
  16. ...and 5 more sections

Figures (4)

  • Figure 1: Main difference between RoPE (left) and CRoPE (right): the latter has a smaller function space despite having same number of parameters in the embedding
  • Figure 2: Main difference between RoPE (left) and CRoPE (right): the latter has a smaller function space despite having same number of parameters in the embedding
  • Figure 3: Illustrative task of token-dependent position attending, with the input shown on the left and desired attention weights on the right. Depending on the token value, the desired attention weights focus on the $i+1$-th token (a) and the $i+2$-th token, respectively.
  • Figure 4: Validation loss curve (top panel) and training loss curve (bottom panel) for different model architecture. Each column corresponds to a dataset.