Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluation of sentence embeddings in downstream and linguistic probing tasks

Christian S. Perone, Roberto Silveira, Thomas S. Paula

TL;DR

The paper conducts a comprehensive benchmark of recent sentence embedding methods across a broad suite of downstream and linguistic probing tasks. It finds that a simple bag-of-words representation built on context-aware embeddings like ELMo can outperform several sentence encoders trained on entailment data, though no method is universally superior. Probing tasks reveal that multi-layer language-model representations capture a wide range of linguistic properties, while task-tuned encoders excel on tasks resembling their training signal. The work suggests that combining language-model context with sentence-level encoding could move toward more universal transfer capabilities while highlighting the value of probing for interpretability.

Abstract

Despite the fast developmental pace of new sentence embedding methods, it is still challenging to find comprehensive evaluations of these different techniques. In the past years, we saw significant improvements in the field of sentence embeddings and especially towards the development of universal sentence encoders that could provide inductive transfer to a wide variety of downstream tasks. In this work, we perform a comprehensive evaluation of recent methods using a wide variety of downstream and linguistic feature probing tasks. We show that a simple approach using bag-of-words with a recently introduced language model for deep context-dependent word embeddings proved to yield better results in many tasks when compared to sentence encoders trained on entailment datasets. We also show, however, that we are still far away from a universal encoder that can perform consistently across several downstream tasks.

Evaluation of sentence embeddings in downstream and linguistic probing tasks

TL;DR

The paper conducts a comprehensive benchmark of recent sentence embedding methods across a broad suite of downstream and linguistic probing tasks. It finds that a simple bag-of-words representation built on context-aware embeddings like ELMo can outperform several sentence encoders trained on entailment data, though no method is universally superior. Probing tasks reveal that multi-layer language-model representations capture a wide range of linguistic properties, while task-tuned encoders excel on tasks resembling their training signal. The work suggests that combining language-model context with sentence-level encoding could move toward more universal transfer capabilities while highlighting the value of probing for interpretability.

Abstract

Despite the fast developmental pace of new sentence embedding methods, it is still challenging to find comprehensive evaluations of these different techniques. In the past years, we saw significant improvements in the field of sentence embeddings and especially towards the development of universal sentence encoders that could provide inductive transfer to a wide variety of downstream tasks. In this work, we perform a comprehensive evaluation of recent methods using a wide variety of downstream and linguistic feature probing tasks. We show that a simple approach using bag-of-words with a recently introduced language model for deep context-dependent word embeddings proved to yield better results in many tasks when compared to sentence encoders trained on entailment datasets. We also show, however, that we are still far away from a universal encoder that can perform consistently across several downstream tasks.

Paper Structure

This paper contains 19 sections, 3 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Evaluation tasks
  4. Downstream tasks
  5. Linguistic probing tasks
  6. Methods
  7. Experimental setup
  8. Downstream classification tasks
  9. Semantic relatedness and textual similarity tasks
  10. Information retrieval tasks
  11. Linguistic probing tasks
  12. Experimental results
  13. Downstream classification tasks
  14. Semantic relatedness and textual similarity tasks
  15. Linguistic probing tasks
  16. ...and 4 more sections

Figures (3)

  • Figure 1: Results for the downstream classifications tasks using a MLP. Best viewed in color.
  • Figure 2: Results of the semantic relatedness and textual similarity tasks. Values are the Pearson correlation coefficient for the test sets. Best viewed in color.
  • Figure 3: Graphical results for the linguistic probing tasks using a MLP. Best viewed in color.