Maximizing Signal in Human-Model Preference Alignment
Kelsey Kraus, Margaret Kroll
TL;DR
The paper addresses how to align large-language-model outputs with end-user preferences by treating labeling disagreement as either noise or signal within a spectrum of task subjectivity. It proposes a methodological framework combining replicable annotation environments and science-backed analyses to minimize noise while maximizing signal from subjective judgments. A case study evaluating two guardrails classifiers demonstrates how human judgments can inform model alignment with user expectations, with results highlighting the need for human-centered evaluation in producing trustworthy AI. The work provides guidelines and quantitative tools to integrate human preferences into ML evaluation, emphasizing practical applicability and acknowledging sampling limitations in real-world deployments.
Abstract
The emergence of powerful LLMs has led to a paradigm shift in Natural Language Understanding and Natural Language Generation. The properties that make LLMs so valuable for these tasks -- creativity, ability to produce fluent speech, and ability to quickly and effectively abstract information from large corpora -- also present new challenges to evaluating their outputs. The rush to market has led teams to fall back on quick, cost-effective automatic evaluations which offer value, but do not obviate the need for human judgments in model training and evaluation. This paper argues that in cases in which end users need to agree with the decisions made by ML models -- e.g. in toxicity detection or extraction of main points for summarization -- models should be trained and evaluated on data that represent the preferences of those users. We support this argument by explicating the role of human feedback in labeling and judgment tasks for model training and evaluation. First, we propose methods for disentangling noise from signal in labeling tasks. Then we show that noise in labeling disagreement can be minimized by adhering to proven methodological best practices, while signal can be maximized to play an integral role in model training and evaluation tasks. Finally, we illustrate best practices by providing a case study in which two guardrails classifiers are evaluated using human judgments to align final model behavior to user preferences. We aim for this paper to provide researchers and professionals with guidelines to integrating human judgments into their ML and generative AI evaluation toolkit, particularly when working toward achieving accurate and unbiased features that align with users' needs and expectations.