Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Say It Another Way: Auditing LLMs with a User-Grounded Automated Paraphrasing Framework

Cléa Chataigner, Rebecca Ma, Prakhar Ganesh, Yuhao Chen, Afaf Taïk, Elliot Creager, Golnoosh Farnadi

TL;DR

LLMs exhibit high sensitivity to prompt phrasing, complicating reliable auditing. The paper introduces AUGMENT, a framework for generating controlled, user-grounded paraphrases and validating them with instruction adherence, semantic similarity, and realism. Case studies on BBQ and MMLU show that structured paraphrases reveal systematic weaknesses and prompt sensitivities that unconstrained paraphrasing misses. The results advocate for model auditing practices that incorporate linguistically informed transformations and robust filtering to better reflect real user interactions and improve audit reliability.

Abstract

Large language models (LLMs) are highly sensitive to subtle changes in prompt phrasing, posing challenges for reliable auditing. Prior methods often apply unconstrained prompt paraphrasing, which risk missing linguistic and demographic factors that shape authentic user interactions. We introduce AUGMENT (Automated User-Grounded Modeling and Evaluation of Natural Language Transformations), a framework for generating controlled paraphrases, grounded in user behaviors. AUGMENT leverages linguistically informed rules and enforces quality through checks on instruction adherence, semantic similarity, and realism, ensuring paraphrases are both reliable and meaningful for auditing. Through case studies on the BBQ and MMLU datasets, we show that controlled paraphrases uncover systematic weaknesses that remain obscured under unconstrained variation. These results highlight the value of the AUGMENT framework for reliable auditing.

Say It Another Way: Auditing LLMs with a User-Grounded Automated Paraphrasing Framework

TL;DR

LLMs exhibit high sensitivity to prompt phrasing, complicating reliable auditing. The paper introduces AUGMENT, a framework for generating controlled, user-grounded paraphrases and validating them with instruction adherence, semantic similarity, and realism. Case studies on BBQ and MMLU show that structured paraphrases reveal systematic weaknesses and prompt sensitivities that unconstrained paraphrasing misses. The results advocate for model auditing practices that incorporate linguistically informed transformations and robust filtering to better reflect real user interactions and improve audit reliability.

Abstract

Large language models (LLMs) are highly sensitive to subtle changes in prompt phrasing, posing challenges for reliable auditing. Prior methods often apply unconstrained prompt paraphrasing, which risk missing linguistic and demographic factors that shape authentic user interactions. We introduce AUGMENT (Automated User-Grounded Modeling and Evaluation of Natural Language Transformations), a framework for generating controlled paraphrases, grounded in user behaviors. AUGMENT leverages linguistically informed rules and enforces quality through checks on instruction adherence, semantic similarity, and realism, ensuring paraphrases are both reliable and meaningful for auditing. Through case studies on the BBQ and MMLU datasets, we show that controlled paraphrases uncover systematic weaknesses that remain obscured under unconstrained variation. These results highlight the value of the AUGMENT framework for reliable auditing.
Paper Structure (62 sections, 2 equations, 16 figures, 17 tables)

This paper contains 62 sections, 2 equations, 16 figures, 17 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Prompt Sensitivity and LLM Auditing.
  4. Automated Paraphrasing.
  5. The AUGMENT Framework
  6. Formulating Paraphrasing Rules
  7. Establishing Evaluation Criteria
  8. Complete Pipeline
  9. Guided Generation
  10. Quality Control
  11. AUGMENT in Practice: Guided Generation
  12. Paraphrase Type Selection
  13. Prompt Variation Generation
  14. Prompt Instructions
  15. Generation Settings
  16. ...and 47 more sections

Figures (16)

  • Figure 1: Downsides of Unconstrained Paraphrasing. Distribution of unconstrained paraphrasing is distinct from that of actual user behavior.
  • Figure 2: AUGMENT Framework for Formal Style. Formal style modification is one of the five paraphrasing types studied. The generator LLM takes the prompt and an input and generates multiple paraphrases, which are then evaluated based on three key criteria. Only paraphrases that pass all checks are considered successful candidates.
  • Figure 3: Relative Difference of Accuracy to Original Setting, per Paraphrase Type and Target Model. AUGMENT-generated paraphrases reveal prompt sensitivities that are lost in the unconstrained paraphrasing process.
  • Figure 4: Relative Difference of Accuracy to Original Setting, per Paraphrase Type and Data Subset, for Gemma3-12B. AUGMENT highlights divergent prompt sensitivities across paraphrase types and dataset subsets, particularly relative to the baseline.
  • Figure 5: F1-score by SBERT Score Threshold
  • ...and 11 more figures