Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

What you can cram into a single vector: Probing sentence embeddings for linguistic properties

Alexis Conneau, German Kruszewski, Guillaume Lample, Loïc Barrault, Marco Baroni

TL;DR

This work probes what linguistic attributes are encoded in sentence representations by introducing a diverse set of 10 targeted probing tasks spanning surface, syntactic, and semantic properties. It systematically analyzes multiple encoders (BiLSTM, Gated ConvNet) trained with varied objectives (NMT, NLI, Seq2Tree, SkipThought, AutoEncoder, and untrained baselines), revealing architecture-induced priors and task-dependent linguistic coverage. Key findings show Bag-of-Vectors surprisingly capturing certain sentence-level cues, while architectural choices and training tasks critically shape deeper syntactic and semantic knowledge; WC features correlate strongly with downstream performance, whereas SentLen often diverges. The study provides a public probing suite and connects probing outcomes to downstream capabilities, offering a framework to benchmark and guide the development of linguistically informed universal sentence representations.

Abstract

Although much effort has recently been devoted to training high-quality sentence embeddings, we still have a poor understanding of what they are capturing. "Downstream" tasks, often based on sentence classification, are commonly used to evaluate the quality of sentence representations. The complexity of the tasks makes it however difficult to infer what kind of information is present in the representations. We introduce here 10 probing tasks designed to capture simple linguistic features of sentences, and we use them to study embeddings generated by three different encoders trained in eight distinct ways, uncovering intriguing properties of both encoders and training methods.

What you can cram into a single vector: Probing sentence embeddings for linguistic properties

TL;DR

This work probes what linguistic attributes are encoded in sentence representations by introducing a diverse set of 10 targeted probing tasks spanning surface, syntactic, and semantic properties. It systematically analyzes multiple encoders (BiLSTM, Gated ConvNet) trained with varied objectives (NMT, NLI, Seq2Tree, SkipThought, AutoEncoder, and untrained baselines), revealing architecture-induced priors and task-dependent linguistic coverage. Key findings show Bag-of-Vectors surprisingly capturing certain sentence-level cues, while architectural choices and training tasks critically shape deeper syntactic and semantic knowledge; WC features correlate strongly with downstream performance, whereas SentLen often diverges. The study provides a public probing suite and connects probing outcomes to downstream capabilities, offering a framework to benchmark and guide the development of linguistically informed universal sentence representations.

Abstract

Although much effort has recently been devoted to training high-quality sentence embeddings, we still have a poor understanding of what they are capturing. "Downstream" tasks, often based on sentence classification, are commonly used to evaluate the quality of sentence representations. The complexity of the tasks makes it however difficult to infer what kind of information is present in the representations. We introduce here 10 probing tasks designed to capture simple linguistic features of sentences, and we use them to study embeddings generated by three different encoders trained in eight distinct ways, uncovering intriguing properties of both encoders and training methods.

Paper Structure

This paper contains 22 sections, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Probing tasks
  3. Surface information
  4. Syntactic information
  5. Semantic information
  6. Sentence embedding models
  7. Sentence encoder architectures
  8. BiLSTM-last/max
  9. Gated ConvNet
  10. Training tasks
  11. Training details
  12. Probing task experiments
  13. Baselines
  14. Encoding architectures
  15. Training tasks
  16. ...and 7 more sections

Figures (3)

  • Figure 1: Probing task scores after each training epoch, for NMT and SkipThought. We also report training score evolution: BLEU for NMT; perplexity (PPL) for SkipThought.
  • Figure 2: Spearman correlation matrix between probing and downstream tasks. Correlations based on all sentence embeddings we investigated (more than 40). Cells in gray denote task pairs that are not significantly correlated (after correcting for multiple comparisons).
  • Figure 3: Evolution of probing tasks results wrt. embedding size. The sentence representations are generated by a BiLSTM-max encoder trained on either NLI or NMT En-Fr, with increasing sentence embedding size.