Clifford Algebraic Rotor Embeddings : Maybe embeddings should start to CARE

TL;DR

RoPE extensions to higher dimensions face non-commutativity challenges that can undermine shift-equivariance. The paper introduces QuatRo, which uses quaternion rotors to parameterize rotation axes, and CARE, which extends to Clifford rotors acting on multivectors, unifying and generalizing prior RoPE variants. QuatRo can reproduce Spherical and Mixed RoPE as special cases, while CARE subsumes them and enables higher-dimensional, multivector positional encoding. Preliminary CIFAR100 experiments suggest CARE can outperform spherical or quaternion RoPE, indicating a principled path toward richer and more flexible rotary embeddings for vision and multimodal data. Overall, the work lays a foundation for dimension-agnostic rotary embeddings with potential applications to video and 3D data, while highlighting computational trade-offs and directions for deeper analysis of multivector grade usage.

Abstract

Rotary Positional Embeddings (RoPE) have demonstrated exceptional performance as a positional encoding method, consistently outperforming their baselines. While recent work has sought to extend RoPE to higher-dimensional inputs, many such extensions are non-commutative, thereby forfeiting RoPE's shift-equivariance property. Spherical RoPE is one such non-commutative variant, motivated by the idea of rotating embedding vectors on spheres rather than circles. However, spherical rotations are inherently non-commutative, making the choice of rotation sequence ambiguous. In this work, we explore a quaternion-based approach -- Quaternion Rotary Embeddings (QuatRo) -- in place of Euler angles, leveraging quaternions' ability to represent 3D rotations to parameterize the axes of rotation. We show Mixed RoPE and Spherical RoPE to be special cases of QuatRo. Further, we propose a generalization of QuatRo to Clifford Algebraic Rotary Embeddings (CARE) using geometric algebra. Viewing quaternions as the even subalgebra of Cl(3,0,0), we extend the notion of rotary embeddings from quaternions to Clifford rotors acting on multivectors. This formulation enables two key generalizations: (1) extending rotary embeddings to arbitrary dimensions, and (2) encoding positional information in multivectors of multiple grades, not just vectors. We present preliminary experiments comparing spherical, quaternion, and Clifford-based rotary embeddings.

Figures (3)

• Figure 1: Multiplication tables of quaternions and the even subalgebra of $\mathrm{Cl}(3,0,0)$, illustrating their isomorphism.
• Figure 1: Performance comparison (top-1 accuracy) on CIFAR100 across methods and implementations.
• Figure 2: QuatRo can rotate around any two unlike Spherical RoPE.