What's In My Human Feedback? Learning Interpretable Descriptions of Preference Data

TL;DR

What’s In My Human Feedback? (WIMHF) presents a data-driven, interpretable framework to explain human feedback in model alignment by first extracting measurable preferences from response pairs via a sparse autoencoder, then describing these features in natural language, and finally linking them to expressed preferences with a logistic model. Across seven datasets, WIMHF reveals both dataset-specific and conflicting preferences, showing how response generation and context shape what humans prefer and how reward models may be exploited. The approach achieves substantial signal using only a handful of interpretable features (approximately four active features per example) and aligns with annotator explanations, enabling practical data curation (e.g., boosting safety by relabeling high-risk examples) and annotator-level personalization with data-efficient updates. These insights support principled data-centric preference learning, improve safety and evaluation practices, and offer a controllable pathway for aligning models with diverse human preferences while mitigating reward hacking.

Abstract

Human feedback can alter language models in unpredictable and undesirable ways, as practitioners lack a clear understanding of what feedback data encodes. While prior work studies preferences over certain attributes (e.g., length or sycophancy), automatically extracting relevant features without pre-specifying hypotheses remains challenging. We introduce What's In My Human Feedback? (WIMHF), a method to explain feedback data using sparse autoencoders. WIMHF characterizes both (1) the preferences a dataset is capable of measuring and (2) the preferences that the annotators actually express. Across 7 datasets, WIMHF identifies a small number of human-interpretable features that account for the majority of the preference prediction signal achieved by black-box models. These features reveal a wide diversity in what humans prefer, and the role of dataset-level context: for example, users on Reddit prefer informality and jokes, while annotators in HH-RLHF and PRISM disprefer them. WIMHF also surfaces potentially unsafe preferences, such as that LMArena users tend to vote against refusals, often in favor of toxic content. The learned features enable effective data curation: re-labeling the harmful examples in Arena yields large safety gains (+37%) with no cost to general performance. They also allow fine-grained personalization: on the Community Alignment dataset, we learn annotator-specific weights over subjective features that improve preference prediction. WIMHF provides a human-centered analysis method for practitioners to better understand and use preference data.

Figures (9)

• Figure 1: What's In My Human Feedback enables automated discovery of preferences from feedback data. We first discover measurable preferences: consistent differences within a pair of responses $(r_A, r_B)$, like "emoji usage," learned by a sparse autoencoder (SAE). Regressing the chosen response $y$ on these features yields expressed preferences, like "win-rate is 15% higher with emojis."
• Figure 2: While some preferences are consistent across datasets, many vary significantly, even flipping from preferred in one dataset to dispreferred in others. We exclude any dataset-feature pairs where the feature does not occur with $\ge$5% prevalence. Error bars are bootstrapped 95% CIs.
• Figure 3: Two applications of WIMHF. (a) Data Curation: On Arena, WIMHF finds that annotators prefer when models fulfill harmful (illegal, sexual, etc.) requests instead of refusing; flipping the chosen and rejected responses for up to 1000 examples that activate this feature increases RewardBench2 safety (green) and preserves overall performance (blue). (b) Personalization: On CA, we show that learning annotator-specific coefficients for a subjective feature---paragraphs vs. lists---improves heldout AUC vs. a fixed global model. Actively sampling examples with the largest feature values (blue line) yields more sample-efficient gains than random samples (orange line). Error bars are bootstrapped 95% CIs, resampling instances in (a) and annotators in (b).
• Figure 4: Human preferences are relatively well-explained by a small number of interpretable features, illustrated by the fact that using the SAE features (blue) does not perform substantially worse than an oracle finetuned reward model (grey). Notably, only four features per SAE input are nonzero, on average. Relative to random chance (AUC = 0.5), the SAE achieves 67% of the improvement realized by the reward model. This trend varies by dataset: for example, the interpretable features are highly explanatory on Chatbot Arena (93% of reward model AUC relative to random) and PRISM (77%), but there is a more substantial gap on HH-RLHF (30%), suggesting that some datasets are harder to explain with simple rules. Training a linear classifier on the full 1536-dimensional embeddings, which the SAE is trained on, does not perform much better, averaging 77% of the full reward model.
• Figure 5: Despite not being used in any step of WIMHF, we find that the SAE's learned features often match annotator-written explanations on the CA dataset. Specifically, 59.9% of annotator explanations match at least one of the four most-active SAE features (vs. 33.7% random; $N=\text{5,000}$). Matches are judged by gpt-5-low, with the prompt given in Figure \ref{['fig:prompt_explanation']}.