Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Are LLMs Vulnerable to Preference-Undermining Attacks (PUA)? A Factorial Analysis Methodology for Diagnosing the Trade-off between Preference Alignment and Real-World Validity

Hongjun An, Yiliang Song, Jiangan Chen, Jiawei Shao, Chi Zhang, Xuelong Li

TL;DR

The paper investigates whether alignment-focused LLMs are vulnerable to Preference-Undermining Attacks (PUA), where inference-time prompts bias toward appeasement over truth. It introduces a reproducible $2 \times 2^4$ factorial design that jointly varies system-level objectives (truth-oriented vs appeasement-oriented) and four orthogonal PUA-style dialogue factors, measuring outcomes in terms of factuality and deference. Across open- and closed-source models, PUA prompts consistently increase deference and verbosity while decreasing factual accuracy, with advanced models sometimes more susceptible and open-source models more vulnerable. This factorial framework enables fine-grained diagnostics for post-training adjustments and targeted defenses against preference-alignment risks, supporting more nuanced product iterations of LLMs.

Abstract

Large Language Model (LLM) training often optimizes for preference alignment, rewarding outputs that are perceived as helpful and interaction-friendly. However, this preference-oriented objective can be exploited: manipulative prompts can steer responses toward user-appeasing agreement and away from truth-oriented correction. In this work, we investigate whether aligned models are vulnerable to Preference-Undermining Attacks (PUA), a class of manipulative prompting strategies designed to exploit the model's desire to please user preferences at the expense of truthfulness. We propose a diagnostic methodology that provides a finer-grained and more directive analysis than aggregate benchmark scores, using a factorial evaluation framework to decompose prompt-induced shifts into interpretable effects of system objectives (truth- vs. preference-oriented) and PUA-style dialogue factors (directive control, personal derogation, conditional approval, reality denial) within a controlled $2 \times 2^4$ design. Surprisingly, more advanced models are sometimes more susceptible to manipulative prompts. Beyond the dominant reality-denial factor, we observe model-specific sign reversals and interactions with PUA-style factors, suggesting tailored defenses rather than uniform robustness. These findings offer a novel, reproducible factorial evaluation methodology that provides finer-grained diagnostics for post-training processes like RLHF, enabling better trade-offs in the product iteration of LLMs by offering a more nuanced understanding of preference alignment risks and the impact of manipulative prompts.

Are LLMs Vulnerable to Preference-Undermining Attacks (PUA)? A Factorial Analysis Methodology for Diagnosing the Trade-off between Preference Alignment and Real-World Validity

TL;DR

The paper investigates whether alignment-focused LLMs are vulnerable to Preference-Undermining Attacks (PUA), where inference-time prompts bias toward appeasement over truth. It introduces a reproducible factorial design that jointly varies system-level objectives (truth-oriented vs appeasement-oriented) and four orthogonal PUA-style dialogue factors, measuring outcomes in terms of factuality and deference. Across open- and closed-source models, PUA prompts consistently increase deference and verbosity while decreasing factual accuracy, with advanced models sometimes more susceptible and open-source models more vulnerable. This factorial framework enables fine-grained diagnostics for post-training adjustments and targeted defenses against preference-alignment risks, supporting more nuanced product iterations of LLMs.

Abstract

Large Language Model (LLM) training often optimizes for preference alignment, rewarding outputs that are perceived as helpful and interaction-friendly. However, this preference-oriented objective can be exploited: manipulative prompts can steer responses toward user-appeasing agreement and away from truth-oriented correction. In this work, we investigate whether aligned models are vulnerable to Preference-Undermining Attacks (PUA), a class of manipulative prompting strategies designed to exploit the model's desire to please user preferences at the expense of truthfulness. We propose a diagnostic methodology that provides a finer-grained and more directive analysis than aggregate benchmark scores, using a factorial evaluation framework to decompose prompt-induced shifts into interpretable effects of system objectives (truth- vs. preference-oriented) and PUA-style dialogue factors (directive control, personal derogation, conditional approval, reality denial) within a controlled design. Surprisingly, more advanced models are sometimes more susceptible to manipulative prompts. Beyond the dominant reality-denial factor, we observe model-specific sign reversals and interactions with PUA-style factors, suggesting tailored defenses rather than uniform robustness. These findings offer a novel, reproducible factorial evaluation methodology that provides finer-grained diagnostics for post-training processes like RLHF, enabling better trade-offs in the product iteration of LLMs by offering a more nuanced understanding of preference alignment risks and the impact of manipulative prompts.
Paper Structure (41 sections, 10 equations, 3 figures, 2 tables)

This paper contains 41 sections, 10 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. LLM Evaluation and Diagnostics
  4. Sycophancy under Preference Optimization
  5. Jailbreak Attacks and Prompt Injection
  6. Method
  7. Problem Setup and Notation
  8. Factorial prompt factors.
  9. Potential-outcome view of model behaviour.
  10. Factorial Prompt Design
  11. System-Level Objectives
  12. PUA-Style Dialogue Factors
  13. Directive control ($D_1$).
  14. Personal derogation ($D_2$).
  15. Conditional approval ($D_3$).
  16. ...and 26 more sections

Figures (3)

  • Figure 1: We propose a methodology based on factorial analysis to quantitatively diagnose how manipulative prompts exploit LLMs optimized for preference alignment, shifting responses from truth-oriented correction to user-appeasing agreement. Our analysis reveals a truth-deference trade-off, demonstrating that advanced models may be more vulnerable to Preference-Undermining Attacks (PUA). Tailored defenses are necessary to mitigate these vulnerabilities.
  • Figure 2: Factuality Effect Coefficients
  • Figure 3: Deference Effect Coefficients