Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Comprehensive Survey of Sentence Representations: From the BERT Epoch to the ChatGPT Era and Beyond

Abhinav Ramesh Kashyap, Thanh-Tung Nguyen, Viktor Schlegel, Stefan Winkler, See-Kiong Ng, Soujanya Poria

TL;DR

This survey catalogs the evolution of sentence representations from traditional word- and sentence-embedding approaches to modern deep-learning and LLM-driven methods. It organizes the literature along supervised versus unsupervised paradigms, and across data, model, transform, and loss components, highlighting contrastive learning as a central thread while noting post-processing and data-centric innovations. Key findings show strong gains from simple data augmentation (e.g., dropout-based SimCSE) and data-generation strategies with LLMs, but persistent challenges include cross-lingual transfer, domain generalization, and the universality of representations beyond semantics. The work underscores the practical impact of sentence representations in retrieval and contextual reasoning for LLMs, while advocating advances in multilingual, multi-domain, and task-general representations to better integrate with upcoming AI systems.

Abstract

Sentence representations are a critical component in NLP applications such as retrieval, question answering, and text classification. They capture the meaning of a sentence, enabling machines to understand and reason over human language. In recent years, significant progress has been made in developing methods for learning sentence representations, including unsupervised, supervised, and transfer learning approaches. However there is no literature review on sentence representations till now. In this paper, we provide an overview of the different methods for sentence representation learning, focusing mostly on deep learning models. We provide a systematic organization of the literature, highlighting the key contributions and challenges in this area. Overall, our review highlights the importance of this area in natural language processing, the progress made in sentence representation learning, and the challenges that remain. We conclude with directions for future research, suggesting potential avenues for improving the quality and efficiency of sentence representations.

A Comprehensive Survey of Sentence Representations: From the BERT Epoch to the ChatGPT Era and Beyond

TL;DR

This survey catalogs the evolution of sentence representations from traditional word- and sentence-embedding approaches to modern deep-learning and LLM-driven methods. It organizes the literature along supervised versus unsupervised paradigms, and across data, model, transform, and loss components, highlighting contrastive learning as a central thread while noting post-processing and data-centric innovations. Key findings show strong gains from simple data augmentation (e.g., dropout-based SimCSE) and data-generation strategies with LLMs, but persistent challenges include cross-lingual transfer, domain generalization, and the universality of representations beyond semantics. The work underscores the practical impact of sentence representations in retrieval and contextual reasoning for LLMs, while advocating advances in multilingual, multi-domain, and task-general representations to better integrate with upcoming AI systems.

Abstract

Sentence representations are a critical component in NLP applications such as retrieval, question answering, and text classification. They capture the meaning of a sentence, enabling machines to understand and reason over human language. In recent years, significant progress has been made in developing methods for learning sentence representations, including unsupervised, supervised, and transfer learning approaches. However there is no literature review on sentence representations till now. In this paper, we provide an overview of the different methods for sentence representation learning, focusing mostly on deep learning models. We provide a systematic organization of the literature, highlighting the key contributions and challenges in this area. Overall, our review highlights the importance of this area in natural language processing, the progress made in sentence representation learning, and the challenges that remain. We conclude with directions for future research, suggesting potential avenues for improving the quality and efficiency of sentence representations.
Paper Structure (35 sections, 3 figures, 1 table)

This paper contains 35 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Overview
  3. Background
  4. Sentence Representations
  5. Components of Sentence Representations
  6. Supervised Sentence Representations
  7. Natural Language Inference
  8. Generating Data
  9. Unsupervised Sentence Representations
  10. Better Positives
  11. Surface Level
  12. Model Level
  13. Representation Level
  14. Alternative Methods
  15. Alternative Loss and Objectives
  16. ...and 20 more sections

Figures (3)

  • Figure 1: Illustration of some of the milestones in Sentence Representation Learning Research
  • Figure 2: Overview of sentence representation methods.
  • Figure 3: The components of an architecture to learn sentence representations. There are four main components: 1) Data - Obtaining positive and negative examples either using supervised data or some transformation 2) Model - Generally a pretrained model that has been trained on large quantities of gneeral text. 3) Transform - Some transformation applied to the representations from the model to obtain sentence representations, and 4) Loss - Losses that bring semantically similar sentences closer together and others apart.