Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Thesis proposal: Are We Losing Textual Diversity to Natural Language Processing?

Josef Jon

TL;DR

The ultimate goal is to develop alternatives that do not enforce uniformity in the distribution of statistical properties in the output and that allow for better global planning of the translation, taking into account the intrinsic ambiguity of the translation task.

Abstract

This thesis argues that the currently widely used Natural Language Processing algorithms possibly have various limitations related to the properties of the texts they handle and produce. With the wide adoption of these tools in rapid progress, we must ask what these limitations are and what are the possible implications of integrating such tools even more deeply into our daily lives. As a testbed, we have chosen the task of Neural Machine Translation (NMT). Nevertheless, we aim for general insights and outcomes, applicable even to current Large Language Models (LLMs). We ask whether the algorithms used in NMT have inherent inductive biases that are beneficial for most types of inputs but might harm the processing of untypical texts. To explore this hypothesis, we define a set of measures to quantify text diversity based on its statistical properties, like uniformity or rhythmicity of word-level surprisal, on multiple scales (sentence, discourse, language). We then conduct a series of experiments to investigate whether NMT systems struggle with maintaining the diversity of such texts, potentially reducing the richness of the language generated by these systems, compared to human translators. We search for potential causes of these limitations rooted in training objectives and decoding algorithms. Our ultimate goal is to develop alternatives that do not enforce uniformity in the distribution of statistical properties in the output and that allow for better global planning of the translation, taking into account the intrinsic ambiguity of the translation task.

Thesis proposal: Are We Losing Textual Diversity to Natural Language Processing?

TL;DR

The ultimate goal is to develop alternatives that do not enforce uniformity in the distribution of statistical properties in the output and that allow for better global planning of the translation, taking into account the intrinsic ambiguity of the translation task.

Abstract

This thesis argues that the currently widely used Natural Language Processing algorithms possibly have various limitations related to the properties of the texts they handle and produce. With the wide adoption of these tools in rapid progress, we must ask what these limitations are and what are the possible implications of integrating such tools even more deeply into our daily lives. As a testbed, we have chosen the task of Neural Machine Translation (NMT). Nevertheless, we aim for general insights and outcomes, applicable even to current Large Language Models (LLMs). We ask whether the algorithms used in NMT have inherent inductive biases that are beneficial for most types of inputs but might harm the processing of untypical texts. To explore this hypothesis, we define a set of measures to quantify text diversity based on its statistical properties, like uniformity or rhythmicity of word-level surprisal, on multiple scales (sentence, discourse, language). We then conduct a series of experiments to investigate whether NMT systems struggle with maintaining the diversity of such texts, potentially reducing the richness of the language generated by these systems, compared to human translators. We search for potential causes of these limitations rooted in training objectives and decoding algorithms. Our ultimate goal is to develop alternatives that do not enforce uniformity in the distribution of statistical properties in the output and that allow for better global planning of the translation, taking into account the intrinsic ambiguity of the translation task.
Paper Structure (25 sections, 4 equations, 9 figures, 4 tables)

This paper contains 25 sections, 4 equations, 9 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Problems and proposed solutions
  3. Measuring diversity
  4. Diversity in MT
  5. Causes of excessive uniformity
  6. Alternative decoding algorithm
  7. Alternative training objective
  8. Objectives
  9. Characteristics of NMT-produced language
  10. Lexical diversity in NMT
  11. Skewed word frequencies
  12. Decoding Algorithms
  13. Training objective
  14. Progress so far
  15. O1: Information distribution in language
  16. ...and 10 more sections

Figures (9)

  • Figure 1: Usage of words that are often produced by ChatGPT in ICLR reviews in years 2020 to 2024. Increased frequency of these words in 2024 shows the prevalence of using ChatGPT for writing reviews. From liang2024monitoring.
  • Figure 2: Proposed text properties that we plan to observe during the MT process.
  • Figure 3: Illustration of the current NMT process and the points addressed in this thesis (red arrows).
  • Figure 4: Surprisal behavior for the two examples sentences, measured by GPT-2 model.
  • Figure 5: Behavior of SLOR measure, depending on power $k$ used in the calculation.
  • ...and 4 more figures