Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating the Smooth Control of Attribute Intensity in Text Generation with LLMs

Shang Zhou, Feng Yao, Chengyu Dong, Zihan Wang, Jingbo Shang

TL;DR

This work tackles the problem of smoothly controlling attribute intensity in text generated by large language models. It defines smooth controllable text generation (SCTG), introduces an automatic evaluation pipeline based on Elo ratings and GPT-4 judgments, and builds a 1,500-query benchmark across five attributes. It compares two training-free methods—prompting with semantic shifters and Representation Engineering (RepE)—across multiple models, finding that prompting is often nearly as effective as RepE and that larger model sizes can hinder smooth control. The framework and findings provide practical guidance for deploying finely calibrated attribute control in real-world LLM applications and establish a scalable evaluation toolkit for SCTG.

Abstract

Controlling the attribute intensity of text generation is crucial across scenarios (e.g., writing conciseness, chatting emotion, and explanation clarity). The remarkable capabilities of large language models (LLMs) have revolutionized text generation, prompting us to explore such \emph{smooth control} of LLM generation. Specifically, we propose metrics to assess the range, calibration, and consistency of the generated text's attribute intensity in response to varying control values, as well as its relevance to the intended context. To quantify the attribute intensity and context relevance, we propose an effective evaluation framework leveraging the Elo rating system and GPT4, both renowned for their robust alignment with human judgment. We look into two viable training-free methods for achieving smooth control of LLMs: (1) Prompting with semantic shifters, and (2) Modifying internal model representations. The evaluations of these two methods are conducted on $5$ different attributes with various models. Our code and dataset can be obtained from \url{https://github.com/ShangDataLab/Smooth-Control}.

Evaluating the Smooth Control of Attribute Intensity in Text Generation with LLMs

TL;DR

This work tackles the problem of smoothly controlling attribute intensity in text generated by large language models. It defines smooth controllable text generation (SCTG), introduces an automatic evaluation pipeline based on Elo ratings and GPT-4 judgments, and builds a 1,500-query benchmark across five attributes. It compares two training-free methods—prompting with semantic shifters and Representation Engineering (RepE)—across multiple models, finding that prompting is often nearly as effective as RepE and that larger model sizes can hinder smooth control. The framework and findings provide practical guidance for deploying finely calibrated attribute control in real-world LLM applications and establish a scalable evaluation toolkit for SCTG.

Abstract

Controlling the attribute intensity of text generation is crucial across scenarios (e.g., writing conciseness, chatting emotion, and explanation clarity). The remarkable capabilities of large language models (LLMs) have revolutionized text generation, prompting us to explore such \emph{smooth control} of LLM generation. Specifically, we propose metrics to assess the range, calibration, and consistency of the generated text's attribute intensity in response to varying control values, as well as its relevance to the intended context. To quantify the attribute intensity and context relevance, we propose an effective evaluation framework leveraging the Elo rating system and GPT4, both renowned for their robust alignment with human judgment. We look into two viable training-free methods for achieving smooth control of LLMs: (1) Prompting with semantic shifters, and (2) Modifying internal model representations. The evaluations of these two methods are conducted on different attributes with various models. Our code and dataset can be obtained from \url{https://github.com/ShangDataLab/Smooth-Control}.
Paper Structure (39 sections, 6 equations, 6 figures, 8 tables)

This paper contains 39 sections, 6 equations, 6 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Controllable Text Generation
  4. Text Style Transfer
  5. Problem Formulation
  6. Definition of Smooth Control
  7. Benchmark Data Construction
  8. Attribute Selection.
  9. Query Generation.
  10. Evaluating Smooth Control
  11. Rating System
  12. Human evaluation of the rating system
  13. Speed-up of Elo Calculations
  14. Metrics
  15. Experiments Setup
  16. ...and 24 more sections

Figures (6)

  • Figure 1: A demonstration for the smooth control of the understandability attribute in the concept explanation scenario, where the control values enable the continuous adjustment of response professionalism, highlighting the nuanced customization of communication.
  • Figure 2: In our quantitative study, we determine the percentage of human preference for pairs of sentences with varying Elo ratings, as assessed through annotations by GPT-4 or GPT-3.5. Additionally, we present the theoretical win probability as defined by the Elo rating algorithm.
  • Figure 3: Comparison of convergence speeds of four different strategies on calculating the Elo ratings.
  • Figure 4: Comparisons between prompting with universal and selected semantic shifters. The Y axis is the attribute intensity. The black dashed lines are the ideal correlation between the control value and the attribute intensity.
  • Figure 5: Examples for human evaluation.
  • ...and 1 more figures