Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Steering Embedding Models with Geometric Rotation: Mapping Semantic Relationships Across Languages and Models

Michael Freenor, Lauren Alvarez

TL;DR

The paper tackles the interpretability gap in modern multilingual embeddings by proposing Rotor-Invariant Shift Estimation (RISE), a rotor-based, geodesic method that represents discourse-level semantic transformations as rotations on the unit hypersphere $\mathbb{S}^{d-1}$. The approach uses a three-step pipeline—canonicalization, prototype learning in tangent space, and geodesic-based prediction—to capture consistent geometric transformations across languages and embedding architectures. Empirical results show strong cross-language transfer for negation (mean ~0.788) and high stability for conditionality, with politeness displaying more language-dependent variability; cross-model transfer is feasible via Morris mappings but depends on language, revealing an English-centric bias in transfer performance. These findings extend the Linear Representation Hypothesis to multilingual, discourse-level semantics and provide a principled geometric framework for interpretable and controllable multilingual embeddings with potential practical impact on cross-lingual NLP applications.

Abstract

Understanding how language and embedding models encode semantic relationships is fundamental to model interpretability and control. While early word embeddings exhibited intuitive vector arithmetic (''king'' - ''man'' + ''woman'' = ''queen''), modern high-dimensional text representations lack straightforward interpretable geometric properties. We introduce Rotor-Invariant Shift Estimation (RISE), a geometric approach that represents semantic transformations as consistent rotational operations in embedding space, leveraging the manifold structure of modern language representations. RISE operations have the ability to operate across both languages and models with high transfer of performance, suggesting the existence of analogous cross-lingual geometric structure. We evaluate RISE across three embedding models, three datasets, and seven morphologically diverse languages in five major language groups. Our results demonstrate that RISE consistently maps discourse-level semantic transformations with distinct grammatical features (e.g., negation and conditionality) across languages and models. This work provides the first systematic demonstration that discourse-level semantic transformations correspond to consistent geometric operations in multilingual embedding spaces, empirically supporting the Linear Representation Hypothesis at the sentence level.

Steering Embedding Models with Geometric Rotation: Mapping Semantic Relationships Across Languages and Models

TL;DR

The paper tackles the interpretability gap in modern multilingual embeddings by proposing Rotor-Invariant Shift Estimation (RISE), a rotor-based, geodesic method that represents discourse-level semantic transformations as rotations on the unit hypersphere . The approach uses a three-step pipeline—canonicalization, prototype learning in tangent space, and geodesic-based prediction—to capture consistent geometric transformations across languages and embedding architectures. Empirical results show strong cross-language transfer for negation (mean ~0.788) and high stability for conditionality, with politeness displaying more language-dependent variability; cross-model transfer is feasible via Morris mappings but depends on language, revealing an English-centric bias in transfer performance. These findings extend the Linear Representation Hypothesis to multilingual, discourse-level semantics and provide a principled geometric framework for interpretable and controllable multilingual embeddings with potential practical impact on cross-lingual NLP applications.

Abstract

Understanding how language and embedding models encode semantic relationships is fundamental to model interpretability and control. While early word embeddings exhibited intuitive vector arithmetic (''king'' - ''man'' + ''woman'' = ''queen''), modern high-dimensional text representations lack straightforward interpretable geometric properties. We introduce Rotor-Invariant Shift Estimation (RISE), a geometric approach that represents semantic transformations as consistent rotational operations in embedding space, leveraging the manifold structure of modern language representations. RISE operations have the ability to operate across both languages and models with high transfer of performance, suggesting the existence of analogous cross-lingual geometric structure. We evaluate RISE across three embedding models, three datasets, and seven morphologically diverse languages in five major language groups. Our results demonstrate that RISE consistently maps discourse-level semantic transformations with distinct grammatical features (e.g., negation and conditionality) across languages and models. This work provides the first systematic demonstration that discourse-level semantic transformations correspond to consistent geometric operations in multilingual embedding spaces, empirically supporting the Linear Representation Hypothesis at the sentence level.

Paper Structure

This paper contains 56 sections, 3 theorems, 10 equations, 11 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Linear Representation Hypothesis
  4. Linear & Geometric Representation Techniques
  5. Steering Vectors & Embedding Models
  6. Challenges in Generalization and Reliability
  7. Theoretical Motivation
  8. Rotor-Invariant Shift Estimation (RISE)
  9. The Rotor Alignment Algorithm
  10. Differentiation from Related Work
  11. Experimental Design
  12. Discourse-level Semantic Changes & Language Selection
  13. Datasets & Embedding Models
  14. Results
  15. Cross-Language Transfer Comparison
  16. ...and 41 more sections

Key Result

Lemma 1

For $n\in\mathbb{S}^{d-1}$, tangent vector $\xi\in T_n\mathbb{S}^{d-1}$, and point $v\in\mathbb{S}^{d-1}\setminus\{-n\}$,

Figures (11)

  • Figure 1: Embedding model heatmap cross-lingual transfer comparison on negation.
  • Figure 2: Cross-language transfer heatmaps for text-embedding-3-large model showing RISE performance across all language pairs for conditionality, negation, and politeness transformations. Darker colors indicate higher cosine similarity between predicted and target embeddings.
  • Figure 3: Cross-Model Semantic Transfer: text-embedding-3-large → bge-m3. Each cell shows transfer performance from source language prototype (text-embedding-3-large) to target language test set (bge-m3). Diagonal elements represent pure cross-model transfer, while off-diagonal elements show combined cross-model and cross-language transfer using Morris statistical mapping morris2020linearity.
  • Figure 4: Cross-language transfer heatmaps for bge-m3 model showing RISE performance across all language pairs for conditionality, negation, and politeness transformations. Darker colors indicate higher cosine similarity between predicted and target embeddings.
  • Figure 5: Cross-language transfer heatmaps for text-embedding-3-large model showing RISE performance across all language pairs for conditionality, negation, and politeness transformations. Darker colors indicate higher cosine similarity between predicted and target embeddings.
  • ...and 6 more figures

Theorems & Definitions (5)

  • Lemma 1: Exponential and logarithmic maps on the unit sphere
  • proof
  • Theorem A.1: RISE commutativity to first order
  • proof
  • Proposition A.1: Per-transformation complexity