"Seeing the Big through the Small": Can LLMs Approximate Human Judgment Distributions on NLI from a Few Explanations?

TL;DR

Human label variation in NLI provides informative signals that are challenging to scale. The authors show that a small number of expert explanations can steer LLMs to produce model judgment distributions (MJDs) closer to human judgment distributions (HJDs), and they demonstrate that MJDs can be used to fine-tune smaller models, though results depend on global distribution shape rather than per-instance distance. By combining MCQA-style prompts, first-token probability extraction, and bias-mitigation strategies (serial vs parallel processing and permutation averaging), they reveal that distance correlation—a global metric—better predicts downstream performance than traditional instance-level divergence measures. The work suggests that explanation-informed MJDs offer a scalable path to capturing HLV across NLP tasks and potentially beyond NLI, with implications for evaluation, annotation efficiency, and task-generalizable methods.

Abstract

Human label variation (HLV) is a valuable source of information that arises when multiple human annotators provide different labels for valid reasons. In Natural Language Inference (NLI) earlier approaches to capturing HLV involve either collecting annotations from many crowd workers to represent human judgment distribution (HJD) or use expert linguists to provide detailed explanations for their chosen labels. While the former method provides denser HJD information, obtaining it is resource-intensive. In contrast, the latter offers richer textual information but it is challenging to scale up to many human judges. Besides, large language models (LLMs) are increasingly used as evaluators ("LLM judges") but with mixed results, and few works aim to study HJDs. This study proposes to exploit LLMs to approximate HJDs using a small number of expert labels and explanations. Our experiments show that a few explanations significantly improve LLMs' ability to approximate HJDs with and without explicit labels, thereby providing a solution to scale up annotations for HJD. However, fine-tuning smaller soft-label aware models with the LLM-generated model judgment distributions (MJDs) presents partially inconsistent results: while similar in distance, their resulting fine-tuned models and visualized distributions differ substantially. We show the importance of complementing instance-level distance measures with a global-level shape metric and visualization to more effectively evaluate MJDs against human judgment distributions.

Figures (9)

• Figure 1: Comparison between approaches to investigate HLV in NLI. Experts first explain the sample individually and then select a label, while crowd workers only record their choices. Explanations provide details for labels to understand HLV. However, it is not clear how to use explanations effectively to model HLV.
• Figure 2: The overall structure of our LLM approximation system. Explanations from 4 annotators in VariErr NLI DBLP:journals/corr/abs-2403-01931 are transformed with corresponding NLI samples together into multiple-choice questions, and the generated soft labels (model judgment distributions) are compared with human judgment distributions from 100 crowed workers in Chaos NLI DBLP:conf/emnlp/NieZB20. Two SOTA open-source LLMs, Mixtral DBLP:journals/corr/abs-2401-04088, and Llama3 llama3, interpret the explanations, and we conduct comparisons on distribution and fine-tuning.
• Figure 3: Distribution comparison results. "n in one" denotes the way LLMs process $n$ explanations at a time.
• Figure 4: Visualization of distributions in ternary plot. Each point represents one of the 341 samples.
• Figure 5: Zooming in (scale=3.3) on Llama3 MJD.