Contextualized Evaluations: Judging Language Model Responses to Underspecified Queries

TL;DR

This work tackles the challenge of evaluating language models on underspecified queries by introducing contextualized evaluations, a protocol that synthetically adds context via follow-up questions and answers. By analyzing large benchmark datasets, the authors show that underspecification is prevalent and can distort evaluation outcomes; they demonstrate that providing context improves evaluator agreement, can flip model rankings, and reduces reliance on surface cues. The study also investigates default model biases toward WEIRD contexts and models’ varying ability to adapt to different contextual attributes, highlighting the need for fine-grained context-aware evaluation. Overall, contextualized evaluations offer a practical, modular approach to better capture model behavior across diverse user contexts, with implications for more reliable benchmarking and fairer assessment of instruction-following and personalization capabilities.

Abstract

Language model users often issue queries that lack specification, where the context under which a query was issued -- such as the user's identity, the query's intent, and the criteria for a response to be useful -- is not explicit. For instance, a good response to a subjective query like "What book should I read next?" would depend on the user's preferences, and a good response to an open-ended query like "How do antibiotics work against bacteria?" would depend on the user's expertise. This makes evaluation of responses to such queries an ill-posed task, as evaluators may make arbitrary judgments about the response quality. To remedy this, we present contextualized evaluations, a protocol that synthetically constructs context surrounding an underspecified query and provides it during evaluation. We find that the presence of context can 1) alter conclusions drawn from evaluation, even flipping benchmark rankings between model pairs, 2) nudge evaluators to make fewer judgments based on surface-level criteria, like style, and 3) provide new insights about model behavior across diverse contexts. Specifically, our procedure suggests a potential bias towards WEIRD (Western, Educated, Industrialized, Rich and Democratic) contexts in models' "default" responses and we find that models are not equally sensitive to following different contexts, even when they are provided in prompts.

Figures (8)

• Figure 1: Queries issued by users to language models are often underspecified and can lead to arbitrary evaluation judgments of response quality. We present contextualized evaluations, where queries are supplemented with surrounding context during evaluation.
• Figure 2: Human validation shows that most generated follow-up questions are important for clarifying an underspecified query and most generated answer sets are realistic, complete and diverse.
• Figure 3: Our work defines two new evaluation settings---adaptive evaluation and implicit context discovery---distinctive from the standard evaluation paradigm which is context-agnostic.
• Figure 4: Types of human and autorater justifications across all evaluations settings. Note that there is a lower percentage of justifications based on surface-level criteria for the context-aware evaluation settings.
• Figure 5: Relevance ratings of default responses from GPT-4o across various contextual attributes, as rated by Gemini-1.5-Pro. These plots suggest that default GPT-4o responses are better aligned towards users from western cultural contexts, high-income individuals and young and middle-aged adults.