Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CreditAudit: 2D Auditing for LLM Evaluation and Selection

Yiliang Song, Hongjun An, Jiangong Xiao, Haofei Zhao, Jiawei Shao, Xuelong Li

TL;DR

CreditAudit reframes LLM selection as a risk-aware problem by treating the interaction protocol as a deployment factor. It systematically samples semantically aligned, non-adversarial system-prompt templates across benchmarks to produce a two-dimensional audit: mean ability $\mu$ and scenario-induced fluctuation $\sigma$, with $\sigma$ mapped to credit grades AAA–BBB for regime-specific prioritization. The approach emphasizes fixed evaluation subsets, alignment of template indices across benchmarks, parsing-based scoring for auditability, and diagnostics to mitigate template drift. Experiments on GPQA, TruthfulQA, and MMLU-Pro reveal that similar $\mu$ can hide divergent $\sigma$, and that robustness can overturn single-score decisions in agentic workflows, advocating a practical rule: minimize $\sigma$ before maximizing $\mu$ within an acceptable grade. The work provides an open-source toolkit for scenario construction, evaluation, and reporting to enable reproducible, deployment-oriented audits and future extensions to broader benchmarks and tasks.

Abstract

Leaderboard scores on public benchmarks have been steadily rising and converging, with many frontier language models now separated by only marginal differences. However, these scores often fail to match users' day to day experience, because system prompts, output protocols, and interaction modes evolve under routine iteration, and in agentic multi step pipelines small protocol shifts can trigger disproportionate failures, leaving practitioners uncertain about which model to deploy. We propose CreditAudit, a deployment oriented credit audit framework that evaluates models under a family of semantically aligned and non adversarial system prompt templates across multiple benchmarks, reporting mean ability as average performance across scenarios and scenario induced fluctuation sigma as a stability risk signal, and further mapping volatility into interpretable credit grades from AAA to BBB via cross model quantiles with diagnostics that mitigate template difficulty drift. Controlled experiments on GPQA, TruthfulQA, and MMLU Pro show that models with similar mean ability can exhibit substantially different fluctuation, and stability risk can overturn prioritization decisions in agentic or high failure cost regimes. By providing a 2D and grade based language for regime specific selection, CreditAudit supports tiered deployment and more disciplined allocation of testing and monitoring effort, enabling more objective and trustworthy model evaluation for real world use.

CreditAudit: 2D Auditing for LLM Evaluation and Selection

TL;DR

CreditAudit reframes LLM selection as a risk-aware problem by treating the interaction protocol as a deployment factor. It systematically samples semantically aligned, non-adversarial system-prompt templates across benchmarks to produce a two-dimensional audit: mean ability and scenario-induced fluctuation , with mapped to credit grades AAA–BBB for regime-specific prioritization. The approach emphasizes fixed evaluation subsets, alignment of template indices across benchmarks, parsing-based scoring for auditability, and diagnostics to mitigate template drift. Experiments on GPQA, TruthfulQA, and MMLU-Pro reveal that similar can hide divergent , and that robustness can overturn single-score decisions in agentic workflows, advocating a practical rule: minimize before maximizing within an acceptable grade. The work provides an open-source toolkit for scenario construction, evaluation, and reporting to enable reproducible, deployment-oriented audits and future extensions to broader benchmarks and tasks.

Abstract

Leaderboard scores on public benchmarks have been steadily rising and converging, with many frontier language models now separated by only marginal differences. However, these scores often fail to match users' day to day experience, because system prompts, output protocols, and interaction modes evolve under routine iteration, and in agentic multi step pipelines small protocol shifts can trigger disproportionate failures, leaving practitioners uncertain about which model to deploy. We propose CreditAudit, a deployment oriented credit audit framework that evaluates models under a family of semantically aligned and non adversarial system prompt templates across multiple benchmarks, reporting mean ability as average performance across scenarios and scenario induced fluctuation sigma as a stability risk signal, and further mapping volatility into interpretable credit grades from AAA to BBB via cross model quantiles with diagnostics that mitigate template difficulty drift. Controlled experiments on GPQA, TruthfulQA, and MMLU Pro show that models with similar mean ability can exhibit substantially different fluctuation, and stability risk can overturn prioritization decisions in agentic or high failure cost regimes. By providing a 2D and grade based language for regime specific selection, CreditAudit supports tiered deployment and more disciplined allocation of testing and monitoring effort, enabling more objective and trustworthy model evaluation for real world use.
Paper Structure (50 sections, 20 equations, 7 figures, 13 tables)

This paper contains 50 sections, 20 equations, 7 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Capability Benchmarking and Public Leaderboards
  4. Sensitivity, Uncertainty, and Risk-Aware Model Selection
  5. Preliminaries
  6. What LLM Evaluation Estimates: Partial Observability and Prompt-Induced Reliability
  7. The pretraining distribution is not fully observable.
  8. The interaction space is effectively unbounded.
  9. Capability Under an Open Interaction Protocol Space
  10. Method
  11. CreditAudit Design
  12. Parsing and Benchmark Scores: Auditability and Reproducibility
  13. From One Leaderboard Score to Two Dimensions: Mean Ability and Stability Risk
  14. Credit Grades as Interpretable "Risk Labels"
  15. Experimental
  16. ...and 35 more sections

Figures (7)

  • Figure 1: Overall score vs. scenario-induced fluctuation with median-split quadrants.
  • Figure 2: Model $\times$ scenario heatmap of overall scores $S_{m,t}$.
  • Figure 3: Across-model mean score $\bar{S}_t$ by scenario (template). A near-flat trend indicates that scenario variants do not introduce a strong systematic shift in overall difficulty across models.
  • Figure 4: Per-model distributions of overall scores across scenarios. Boxplots reveal not only average and variance but also tail behavior and scenario-specific collapses, complementing summary statistics $(\mu_m,\sigma_m)$.
  • Figure 5: GPQA score-fluctuation map using $(\mu_{m,\mathrm{GPQA}}, \sigma_{m,\mathrm{GPQA}})$. Center markers show per-model means and fluctuations; semi-transparent clouds show per-scenario realizations, highlighting scenario sensitivity on reasoning-intensive items.
  • ...and 2 more figures