DICE: A Framework for Dimensional and Contextual Evaluation of Language Models

TL;DR

DICE addresses the gap between traditional benchmark-based LM evaluation and real-world deployment by proposing a Dimensional and Contextual Evaluation framework that partitions LM behavior into context-agnostic and context-specific dimensions. It outlines how to operationalize measurements, including metric selection, stakeholder-driven weighting, and the use of context-aligned datasets, to produce actionable comparisons for deployment decisions. Through mental healthcare and education case studies, the paper demonstrates how domain constraints shape desirable LM behavior and how DICE can guide context-driven evaluation. While offering opportunities for more interpretable and stakeholder-aligned assessments, the authors also discuss challenges in data availability, framework adaptability, and multi-objective optimization, positioning DICE as a practical precursor to broader empirical studies. Overall, DICE aims to render LM evaluation more context-sensitive, interpretable, and decision-relevant for diverse real-world applications.

Abstract

Language models (LMs) are increasingly being integrated into a wide range of applications, yet the modern evaluation paradigm does not sufficiently reflect how they are actually being used. Current evaluations rely on benchmarks that often lack direct applicability to the real-world contexts in which LMs are being deployed. To address this gap, we propose Dimensional and Contextual Evaluation (DICE), an approach that evaluates LMs on granular, context-dependent dimensions. In this position paper, we begin by examining the insufficiency of existing LM benchmarks, highlighting their limited applicability to real-world use cases. Next, we propose a set of granular evaluation parameters that capture dimensions of LM behavior that are more meaningful to stakeholders across a variety of application domains. Specifically, we introduce the concept of context-agnostic parameters - such as robustness, coherence, and epistemic honesty - and context-specific parameters that must be tailored to the specific contextual constraints and demands of stakeholders choosing to deploy LMs into a particular setting. We then discuss potential approaches to operationalize this evaluation framework, finishing with the opportunities and challenges DICE presents to the LM evaluation landscape. Ultimately, this work serves as a practical and approachable starting point for context-specific and stakeholder-relevant evaluation of LMs.