Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CAT: A Metric-Driven Framework for Analyzing the Consistency-Accuracy Relation of LLMs under Controlled Input Variations

Paulo Cavalin, Cassia Sanctos, Marcelo Grave, Claudio Pinhanez, Yago Primerano

TL;DR

The paper tackles the lack of tools to evaluate how LLMs balance accuracy with response consistency under input perturbations. It introduces CAT, a framework built around CAR curves and the MCA metric, plus the CORE index to summarize trade-offs, and demonstrates its utility with eight LLMs across four MC benchmarks. CAR curves reveal nuanced behaviors and potential reliability issues not captured by traditional accuracy metrics, guiding model selection for high-stakes applications. The framework is extensible to open-ended tasks via adaptable scoring functions, broadening applicability to real-world evaluation scenarios.

Abstract

We introduce \textsc{CAT}, a framework designed to evaluate and visualize the \emph{interplay} of \emph{accuracy} and \emph{response consistency} of Large Language Models (LLMs) under controllable input variations, using multiple-choice (MC) benchmarks as a case study. Current evaluation practices primarily focus on model capabilities such as accuracy or benchmark scores and, more recently, measuring consistency is being considered an essential property for deploying LLMs in high-stake, real-world applications. We argue in this paper that although both dimensions should still be evaluated independently, their inter-dependency also need to be considered for a more nuanced evaluation of LLMs. At the core of \textsc{CAT} are the \emph{Consistency-Accuracy Relation (CAR)} curves, which visualize how model accuracy varies with increasing consistency requirements, as defined by the \emph{Minimum-Consistency Accuracy (MCA)} metric. We further propose the \emph{Consistency-Oriented Robustness Estimate (CORE)} index, a global metric that combines the area and shape of the CAR curve to quantify the trade-off between accuracy and consistency. We present a practical demonstration of our framework across a diverse set of generalist and domain-specific LLMs, evaluated on multiple MC benchmarks. We also outline how \textsc{CAT} can be extended beyond MC tasks to support long-form, open-ended evaluations through adaptable scoring functions.

CAT: A Metric-Driven Framework for Analyzing the Consistency-Accuracy Relation of LLMs under Controlled Input Variations

TL;DR

The paper tackles the lack of tools to evaluate how LLMs balance accuracy with response consistency under input perturbations. It introduces CAT, a framework built around CAR curves and the MCA metric, plus the CORE index to summarize trade-offs, and demonstrates its utility with eight LLMs across four MC benchmarks. CAR curves reveal nuanced behaviors and potential reliability issues not captured by traditional accuracy metrics, guiding model selection for high-stakes applications. The framework is extensible to open-ended tasks via adaptable scoring functions, broadening applicability to real-world evaluation scenarios.

Abstract

We introduce \textsc{CAT}, a framework designed to evaluate and visualize the \emph{interplay} of \emph{accuracy} and \emph{response consistency} of Large Language Models (LLMs) under controllable input variations, using multiple-choice (MC) benchmarks as a case study. Current evaluation practices primarily focus on model capabilities such as accuracy or benchmark scores and, more recently, measuring consistency is being considered an essential property for deploying LLMs in high-stake, real-world applications. We argue in this paper that although both dimensions should still be evaluated independently, their inter-dependency also need to be considered for a more nuanced evaluation of LLMs. At the core of \textsc{CAT} are the \emph{Consistency-Accuracy Relation (CAR)} curves, which visualize how model accuracy varies with increasing consistency requirements, as defined by the \emph{Minimum-Consistency Accuracy (MCA)} metric. We further propose the \emph{Consistency-Oriented Robustness Estimate (CORE)} index, a global metric that combines the area and shape of the CAR curve to quantify the trade-off between accuracy and consistency. We present a practical demonstration of our framework across a diverse set of generalist and domain-specific LLMs, evaluated on multiple MC benchmarks. We also outline how \textsc{CAT} can be extended beyond MC tasks to support long-form, open-ended evaluations through adaptable scoring functions.
Paper Structure (13 sections, 10 equations, 5 figures, 1 table)

This paper contains 13 sections, 10 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Notation and Commonly Used Metrics
  4. The CAT Framework
  5. Minimum-Consistency Accuracy (MCA)
  6. Consistency-Accuracy Relation (CAR) Curves
  7. Consistency-Oriented Robustness Estimate (CORE) Index
  8. Empirical Validation of the CAT Framework
  9. Experiment Details
  10. Analysis of CAR Curves and Metric Behavior
  11. Extending CAT to Open-Ended Benchmarks
  12. Conclusion and Future Work
  13. Detailed Results

Figures (5)

  • Figure 1: CAR curves for synthetic models starting from a chance model to models with increasing bias toward the correct answer.
  • Figure 2: Metric value growth across synthetic models. CORE emphasizes early gains and reduces bias towards chance.
  • Figure 3: CAR curves across benchmarks.
  • Figure 4: Metric growth curves, sorted by mean score across all metrics. Legends indicate the slope from chance to best-performing model.
  • Figure 5: Similar to Figure \ref{['fig:metric_value_growth']} but with additional curves for MCA, showing that this metric tends to approximate MV as the consistency parameter decreases.