Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Structure of Relation Decoding Linear Operators in Large Language Models

Miranda Anna Christ, Adrián Csiszárik, Gergely Becsó, Dániel Varga

TL;DR

The paper investigates how transformer models encode relational knowledge through Linear Relational Embeddings (LREs) and shows that a collection of such decoders can be compressed into order-3 tensor networks without substantial loss in decoding accuracy. Using a cross-evaluation protocol, the authors reveal that decoders rely on shared, coarse-grained properties rather than strictly distinct relation mappings, yielding a property-based rather than relation-specific organization. They demonstrate two tensor-network architectures (SimpleOrder3Network and TriangleTensorNetwork) that achieve high compression, analyze the semantic structure via cross-evaluation blocks, and show that generalization to held-out relations is limited to semantically close relations, with strong generalization in a controlled arithmetic dataset. The work connects to broader themes in knowledge representation, tensor decompositions, and model compression, highlighting both interpretability benefits and the boundaries of generalization in real-world language domains.

Abstract

This paper investigates the structure of linear operators introduced in Hernandez et al. [2023] that decode specific relational facts in transformer language models. We extend their single-relation findings to a collection of relations and systematically chart their organization. We show that such collections of relation decoders can be highly compressed by simple order-3 tensor networks without significant loss in decoding accuracy. To explain this surprising redundancy, we develop a cross-evaluation protocol, in which we apply each linear decoder operator to the subjects of every other relation. Our results reveal that these linear maps do not encode distinct relations, but extract recurring, coarse-grained semantic properties (e.g., country of capital city and country of food are both in the country-of-X property). This property-centric structure clarifies both the operators' compressibility and highlights why they generalize only to new relations that are semantically close. Our findings thus interpret linear relational decoding in transformer language models as primarily property-based, rather than relation-specific.

The Structure of Relation Decoding Linear Operators in Large Language Models

TL;DR

The paper investigates how transformer models encode relational knowledge through Linear Relational Embeddings (LREs) and shows that a collection of such decoders can be compressed into order-3 tensor networks without substantial loss in decoding accuracy. Using a cross-evaluation protocol, the authors reveal that decoders rely on shared, coarse-grained properties rather than strictly distinct relation mappings, yielding a property-based rather than relation-specific organization. They demonstrate two tensor-network architectures (SimpleOrder3Network and TriangleTensorNetwork) that achieve high compression, analyze the semantic structure via cross-evaluation blocks, and show that generalization to held-out relations is limited to semantically close relations, with strong generalization in a controlled arithmetic dataset. The work connects to broader themes in knowledge representation, tensor decompositions, and model compression, highlighting both interpretability benefits and the boundaries of generalization in real-world language domains.

Abstract

This paper investigates the structure of linear operators introduced in Hernandez et al. [2023] that decode specific relational facts in transformer language models. We extend their single-relation findings to a collection of relations and systematically chart their organization. We show that such collections of relation decoders can be highly compressed by simple order-3 tensor networks without significant loss in decoding accuracy. To explain this surprising redundancy, we develop a cross-evaluation protocol, in which we apply each linear decoder operator to the subjects of every other relation. Our results reveal that these linear maps do not encode distinct relations, but extract recurring, coarse-grained semantic properties (e.g., country of capital city and country of food are both in the country-of-X property). This property-centric structure clarifies both the operators' compressibility and highlights why they generalize only to new relations that are semantically close. Our findings thus interpret linear relational decoding in transformer language models as primarily property-based, rather than relation-specific.

Paper Structure

This paper contains 66 sections, 8 equations, 9 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background and Notation
  3. Relations in Transformer Language Models
  4. Technical Details
  5. Models and Datasets
  6. Training Tensor Networks to Represent a Collection of Relations
  7. From the naïve representation to tensor networks
  8. Tensor Network Architectures
  9. SimpleOrder3Network
  10. TriangleTensorNetwork
  11. Parameter counts
  12. From matrices to affine maps
  13. Additional embedder network on the relation leg
  14. Training Tensor Networks
  15. End-to-end training of tensor networks
  16. ...and 51 more sections

Figures (9)

  • Figure 1: Compressing relation decoding linear operators with order-3 tensor networks.
  • Figure 2: Cross-evaluation results for the dataset of lre. Each cell shows the faithfulness of a matrix obtained using the row relation and evaluated on the column relation.
  • Figure 3: Cross-evaluation results for the extended dataset. The vertical axis indicates the relation used to obtain the matrix, while the horizontal axis indicates the relation it is tested on. Values represent faithfulness scores.
  • Figure 4: Test faithfulness results on sample-wise generalization on the dataset of lre (left), and relation-wise generalization results on the mathematical dataset (right).
  • Figure 5: Cross-evaluation results for the extended dataset using the Llama-3.1-8Bllama3 model. Each cell of the matrix shows the faithfulness score calculated using the decoder obtained from the row relation, and evaluated on the column relation.
  • ...and 4 more figures

Theorems & Definitions (2)

  • Definition 1: Faithfulness
  • Definition 2: Cross-evaluation protocol