Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LDIR: Low-Dimensional Dense and Interpretable Text Embeddings with Relative Representations

Yile Wang, Zhanyu Shen, Hui Huang

TL;DR

LDIR introduces a low-dimensional, dense yet interpretable text embedding by leveraging relative representations to a set of farthest-point-sampled anchor texts. It replaces high-dimensional 0/1 QA-embeddings with compact floating-point vectors $e_{dense}^{interp}(t)=[Rel(a_1,t),...,Rel(a_n,t)]$ where $n\approx200$–$500$, and $Rel$ is computed via a backbone encoder. Across semantic textual similarity, retrieval, and clustering benchmarks, LDIR achieves competitive performance with black-box embeddings and outperforms existing interpretable baselines while offering substantially reduced dimensionality and cognitive load. The work analyzes anchor selection, anchor-length effects, and fine-grained relatedness, and discusses practical use cases and privacy benefits of relative representations, outlining future avenues to enhance interpretability without sacrificing semantic expressiveness.

Abstract

Semantic text representation is a fundamental task in the field of natural language processing. Existing text embedding (e.g., SimCSE and LLM2Vec) have demonstrated excellent performance, but the values of each dimension are difficult to trace and interpret. Bag-of-words, as classic sparse interpretable embeddings, suffers from poor performance. Recently, Benara et al. (2024) propose interpretable text embeddings using large language models, which forms "0/1" embeddings based on responses to a series of questions. These interpretable text embeddings are typically high-dimensional (larger than 10,000). In this work, we propose Low-dimensional (lower than 500) Dense and Interpretable text embeddings with Relative representations (LDIR). The numerical values of its dimensions indicate semantic relatedness to different anchor texts through farthest point sampling, offering both semantic representation as well as a certain level of traceability and interpretability. We validate LDIR on multiple semantic textual similarity, retrieval, and clustering tasks. Extensive experimental results show that LDIR performs close to the black-box baseline models and outperforms the interpretable embeddings baselines with much fewer dimensions. Code is available at https://github.com/szu-tera/LDIR.

LDIR: Low-Dimensional Dense and Interpretable Text Embeddings with Relative Representations

TL;DR

LDIR introduces a low-dimensional, dense yet interpretable text embedding by leveraging relative representations to a set of farthest-point-sampled anchor texts. It replaces high-dimensional 0/1 QA-embeddings with compact floating-point vectors where –, and is computed via a backbone encoder. Across semantic textual similarity, retrieval, and clustering benchmarks, LDIR achieves competitive performance with black-box embeddings and outperforms existing interpretable baselines while offering substantially reduced dimensionality and cognitive load. The work analyzes anchor selection, anchor-length effects, and fine-grained relatedness, and discusses practical use cases and privacy benefits of relative representations, outlining future avenues to enhance interpretability without sacrificing semantic expressiveness.

Abstract

Semantic text representation is a fundamental task in the field of natural language processing. Existing text embedding (e.g., SimCSE and LLM2Vec) have demonstrated excellent performance, but the values of each dimension are difficult to trace and interpret. Bag-of-words, as classic sparse interpretable embeddings, suffers from poor performance. Recently, Benara et al. (2024) propose interpretable text embeddings using large language models, which forms "0/1" embeddings based on responses to a series of questions. These interpretable text embeddings are typically high-dimensional (larger than 10,000). In this work, we propose Low-dimensional (lower than 500) Dense and Interpretable text embeddings with Relative representations (LDIR). The numerical values of its dimensions indicate semantic relatedness to different anchor texts through farthest point sampling, offering both semantic representation as well as a certain level of traceability and interpretability. We validate LDIR on multiple semantic textual similarity, retrieval, and clustering tasks. Extensive experimental results show that LDIR performs close to the black-box baseline models and outperforms the interpretable embeddings baselines with much fewer dimensions. Code is available at https://github.com/szu-tera/LDIR.
Paper Structure (20 sections, 7 equations, 4 figures, 10 tables)

This paper contains 20 sections, 7 equations, 4 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. QA-Embedding Baseline
  5. "0/1" Embedding to Dense Embedding
  6. Anchor Texts Selection
  7. Overall Comparison
  8. Experiments
  9. Settings
  10. Main Results
  11. Evaluation on Cognitive Load
  12. Analyses and Discussions
  13. Anchor Texts Sampling Methods
  14. The Impact of Length of Anchor Texts
  15. Fine-grained Relatedness Calculation
  16. ...and 5 more sections

Figures (4)

  • Figure 1: Comparison between baselines (top) and our method (bottom). benara2024crafting and sun2024general uses the generated questions and "yes/no" answers to build high-dimensional "0/1" text embeddings. We use anchor texts via farthest point sampling to calculate the relatedness and build low-dimensional dense text embeddings.
  • Figure 2: Example of extracting four anchor texts using farthest point sampling. Each dot represents a text embedding through an encoder, with the blue dots indicating the extracted anchor texts.
  • Figure 3: Comparison of different anchor texts sampling methods (a) and different settings of anchor texts length (b). The horizontal axis represents the different number (20$\sim$800) of different anchor texts.
  • Figure 4: Comparison between the calculation of general (top) and fine-grained (bottom) relatedness scores.