Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CEV-LM: Controlled Edit Vector Language Model for Shaping Natural Language Generations

Samraj Moorjani, Adit Krishnan, Hari Sundaram

TL;DR

CEV-LM is introduced - a lightweight, semi-autoregressive language model that utilizes constrained edit vectors to control three complementary metrics (speed, volume, and circuitousness) that quantify the shape of text and provides significantly more targeted and precise control of these three metrics.

Abstract

As large-scale language models become the standard for text generation, there is a greater need to tailor the generations to be more or less concise, targeted, and informative, depending on the audience/application. Existing control approaches primarily adjust the semantic (e.g., emotion, topics), structural (e.g., syntax tree, parts-of-speech), and lexical (e.g., keyword/phrase inclusion) properties of text, but are insufficient to accomplish complex objectives such as pacing which control the complexity and readability of the text. In this paper, we introduce CEV-LM - a lightweight, semi-autoregressive language model that utilizes constrained edit vectors to control three complementary metrics (speed, volume, and circuitousness) that quantify the shape of text (e.g., pacing of content). We study an extensive set of state-of-the-art CTG models and find that CEV-LM provides significantly more targeted and precise control of these three metrics while preserving semantic content, using less training data, and containing fewer parameters.

CEV-LM: Controlled Edit Vector Language Model for Shaping Natural Language Generations

TL;DR

CEV-LM is introduced - a lightweight, semi-autoregressive language model that utilizes constrained edit vectors to control three complementary metrics (speed, volume, and circuitousness) that quantify the shape of text and provides significantly more targeted and precise control of these three metrics.

Abstract

As large-scale language models become the standard for text generation, there is a greater need to tailor the generations to be more or less concise, targeted, and informative, depending on the audience/application. Existing control approaches primarily adjust the semantic (e.g., emotion, topics), structural (e.g., syntax tree, parts-of-speech), and lexical (e.g., keyword/phrase inclusion) properties of text, but are insufficient to accomplish complex objectives such as pacing which control the complexity and readability of the text. In this paper, we introduce CEV-LM - a lightweight, semi-autoregressive language model that utilizes constrained edit vectors to control three complementary metrics (speed, volume, and circuitousness) that quantify the shape of text (e.g., pacing of content). We study an extensive set of state-of-the-art CTG models and find that CEV-LM provides significantly more targeted and precise control of these three metrics while preserving semantic content, using less training data, and containing fewer parameters.
Paper Structure (24 sections, 4 equations, 3 figures, 13 tables)

This paper contains 24 sections, 4 equations, 3 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Controllable Text Generation
  4. Prototype-then-Edit
  5. Nonstandard Control Conditions
  6. Controllable Edit Vectors
  7. Experimental Settings
  8. Results
  9. Control Evaluation
  10. Semantic Similarity
  11. Lexical Similarity
  12. Tolerance Tuning
  13. Training Delta Distribution
  14. Qualitative Results
  15. Conclusion
  16. ...and 9 more sections

Figures (3)

  • Figure 1: Generated examples of change in speed, volume, and circuitousness, metrics that define the shape of text, and stylized illustrations. The points represent the word embeddings of windows of text, $\{x_1, ..., x_n\}$. The original text has a lower value of the metric, and our generation ($\textsc{Cev-LM}$) demonstrates a higher value.
  • Figure 2: Histogram of delta values (i.e., $s(x) - s(x')$) within the Yelp Restaurant Review Corpus. The x-axis represents the difference in speed within the pairs of our created neighborhood, $\mathcal{N}(x)$, without any constraint on speed. The y-axis counts the number of pairs exhibiting the given delta in log-scale.
  • Figure 3: The number of samples used for training versus percent error. The attribute is denoted by the shape, target delta by the color, and the border indicates that perturbation was used. Despite access to significantly fewer samples, low-resource models exhibit similar amounts of control to high-resource models.