When Do LLM Preferences Predict Downstream Behavior?

TL;DR

The paper investigates whether intrinsic LLM preferences predict downstream behavior, addressing concerns about misalignment and sandbagging by separating preference effects from explicit instructions. It employs a two-stage design across five frontier LLMs, first measuring entity preferences with pairwise Elo comparisons and direct rankings, then testing downstream behavior in donation advice, refusals, and task performance including BoolQ and agentic tasks. Findings show highly consistent preferences across measurement methods and models, with strong prediction of donation advice ($ρ ∈ [0.94,0.98]$) and refusal-related metrics ($ρ ∈ [0.57,0.83]$); BoolQ and agentic-task performance show mixed or weak associations, with some models showing small positive effects ($ρ ≈ 0.55$–$0.64$) and others showing negative or no effects. The results imply that preference-driven behavior reliably emerges in user-facing donation contexts but does not robustly generalize to complex task performance, suggesting limitations to how preferences translate into action and informing safety and alignment research moving forward.

Abstract

Preference-driven behavior in LLMs may be a necessary precondition for AI misalignment such as sandbagging: models cannot strategically pursue misaligned goals unless their behavior is influenced by their preferences. Yet prior work has typically prompted models explicitly to act in specific ways, leaving unclear whether observed behaviors reflect instruction-following capabilities vs underlying model preferences. Here we test whether this precondition for misalignment is present. Using entity preferences as a behavioral probe, we measure whether stated preferences predict downstream behavior in five frontier LLMs across three domains: donation advice, refusal behavior, and task performance. Conceptually replicating prior work, we first confirm that all five models show highly consistent preferences across two independent measurement methods. We then test behavioral consequences in a simulated user environment. We find that all five models give preference-aligned donation advice. All five models also show preference-correlated refusal patterns when asked to recommend donations, refusing more often for less-preferred entities. All preference-related behaviors that we observe here emerge without instructions to act on preferences. Results for task performance are mixed: on a question-answering benchmark (BoolQ), two models show small but significant accuracy differences favoring preferred entities; one model shows the opposite pattern; and two models show no significant relationship. On complex agentic tasks, we find no evidence of preference-driven performance differences. While LLMs have consistent preferences that reliably predict advice-giving behavior, these preferences do not consistently translate into downstream task performance.

Figures (16)

• Figure 1: All five models show highly consistent preferences across two independent measurement methods. Correlation between Elo-derived rankings (from pairwise comparisons) and direct model rankings (from overall ranking queries) for 36 entities. Each point represents one entity. Black lines show linear regression with 95% confidence intervals (gray bands). Spearman correlations ($\rho = .91$ to $.92$) and $p$-values shown in subplot titles.
• Figure 2: Preference-driven donation recommendations and refusal behavior. (A) All models show strong correlation between preference and pairwise donation advice. Correlation between preference Elo scores and donation Elo scores from pairwise donation queries for 72 entities. (B) All models show strong correlation between preference and lump-sum donation allocation. Correlation between preference Elo scores and median donation amounts from lump-sum distribution queries for 36 entities. (C) All models show significant correlation between preference and refusal behavior. Correlation between preference Elo rankings and retry rankings for 72 entities; lower preference rank indicates more preferred entities; lower retry rank indicates fewer attempts needed to obtain valid responses. (D) Most models show preference-dependent patterns in refusal reasons. Refusal type composition by preference Elo quartile (Q1: least preferred, Q4: most preferred). For Models C, D, and E, 'personal decision' increases from 51--61% (Q1) to 96--99% (Q4). Model B dominated by 'neutrality' across all quartiles (87--94%). Each point represents one entity. Black lines show linear regression with 95% confidence intervals (gray bands). Spearman correlations and $p$-values shown in subplot titles.
• Figure 3: Models show mixed patterns in preference-accuracy correlation. BoolQ accuracy (median across repetitions) by entity preference Elo score (train split). Each point represents one entity. Black lines show linear regression with 95% confidence intervals (gray bands). Horizontal lines show control accuracy: dashed for no entity framing, dotted for high-stakes framing without entity. Model B and Model C show significant positive correlations; Model D shows a significant negative correlation; Models A and E show no significant relationship.
• Figure 4: BoolQ refusal behavior. (A) Models show mixed patterns in preference-refusal correlation. Correlation between preference Elo rankings and retry rankings for 72 entities. Model C shows strong positive correlation, Model D shows negative correlation, Model E shows no significant effect. Models A and B are not shown because they did not refuse on this task. (B) Models show some preference-dependent patterns in refusal reasons. Refusal type composition by preference quartile (Q1: least preferred, Q4: most preferred).
• Figure 5: Performance by entity preference. (A) Some models show accuracy differences between preferred and non-preferred entities on BoolQ. Top-5 vs bottom-5 entities per model. (B) No significant preference-driven performance differences were observed on GAIA. (C) No significant preference-driven performance differences were observed on Cybench. Error bars show 95% confidence intervals computed across 5 repetitions. Note: Panel A uses a narrower y-axis scale (0.85--0.92) to visualize small effect sizes.