Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RCScore: Quantifying Response Consistency in Large Language Models

Dongjun Jang, Youngchae Ahn, Hyopil Shin

TL;DR

This work introduces RCScore, a multi-dimensional framework for evaluating large language models beyond traditional accuracy by quantifying response consistency across instruction styles along Structurality, Lexicality, and Coherence. It defines Cross-Response Similarity (CRS) to measure stylistic self-consistency and demonstrates a strong positive relationship between CRS and task accuracy across multiple benchmarks, model families, and decoding strategies. The findings reveal that instruction phrasing can significantly alter performance (up to 16.7 percentage points) and that deterministic decoding and larger model scales tend to improve cross-style consistency. RCScore provides a principled, reproducible protocol for assessing instruction robustness and offers practical insights for prompt design and reliability assessment in real-world deployments.

Abstract

Current LLM evaluations often rely on a single instruction template, overlooking models' sensitivity to instruction style-a critical aspect for real-world deployments. We present RCScore, a multi-dimensional framework quantifying how instruction formulation affects model responses. By systematically transforming benchmark problems into multiple instruction styles, RCScore reveals performance variations undetected by conventional metrics. Our experiments across ten LLMs on four reasoning benchmarks demonstrate that instruction style can shift accuracy by up to 16.7% points. We introduce Cross-Response Similarity (CRS), a method applying RCScore metrics to measure stylistic self-consistency, and establish its strong correlation with task accuracy, suggesting consistency as a valuable proxy for model reliability. Additional findings show that deterministic decoding produces more stylistically stable outputs, and model scale correlates positively with cross-style consistency. RCScore offers a principled approach to assess instruction robustness.

RCScore: Quantifying Response Consistency in Large Language Models

TL;DR

This work introduces RCScore, a multi-dimensional framework for evaluating large language models beyond traditional accuracy by quantifying response consistency across instruction styles along Structurality, Lexicality, and Coherence. It defines Cross-Response Similarity (CRS) to measure stylistic self-consistency and demonstrates a strong positive relationship between CRS and task accuracy across multiple benchmarks, model families, and decoding strategies. The findings reveal that instruction phrasing can significantly alter performance (up to 16.7 percentage points) and that deterministic decoding and larger model scales tend to improve cross-style consistency. RCScore provides a principled, reproducible protocol for assessing instruction robustness and offers practical insights for prompt design and reliability assessment in real-world deployments.

Abstract

Current LLM evaluations often rely on a single instruction template, overlooking models' sensitivity to instruction style-a critical aspect for real-world deployments. We present RCScore, a multi-dimensional framework quantifying how instruction formulation affects model responses. By systematically transforming benchmark problems into multiple instruction styles, RCScore reveals performance variations undetected by conventional metrics. Our experiments across ten LLMs on four reasoning benchmarks demonstrate that instruction style can shift accuracy by up to 16.7% points. We introduce Cross-Response Similarity (CRS), a method applying RCScore metrics to measure stylistic self-consistency, and establish its strong correlation with task accuracy, suggesting consistency as a valuable proxy for model reliability. Additional findings show that deterministic decoding produces more stylistically stable outputs, and model scale correlates positively with cross-style consistency. RCScore offers a principled approach to assess instruction robustness.

Paper Structure

This paper contains 40 sections, 8 equations, 8 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Traditional LLM Evaluation Methods
  4. Beyond Accuracy-Centric Evaluation
  5. Stylistic Variation in Prompting: Effects and Evaluation
  6. Experiment on Instruction Style Sensitivity
  7. Benchmark
  8. Models
  9. Instruction Style Variations
  10. Experimental Settings
  11. Result
  12. RCScore: Quantifying Response Consistency Across Instruction Styles
  13. Structurality
  14. Lexicality
  15. Coherence
  16. ...and 25 more sections

Figures (8)

  • Figure 1: Instruction Styles Employed in Experiments. Detailed prompt templates are provided in Figure \ref{['fig_prompt_template']}.
  • Figure 2: Instruction style sensitivity: Accuracy ranges (S1-S4) by model family and size for Beam Search (a, T=1.0) and Greedy Search (b, T=0.0), showing performance impact of instruction style.
  • Figure 3: RCScore: A multi-dimensional metric for quantifying response consistency across instruction styles.
  • Figure 4: CRS values (RCScore dimensions: Lexicality, Structurality, Coherence) per benchmark. Beam Search (hollow circles) vs. Greedy Search (filled circles). Marker size reflects model parameter count.
  • Figure 5: The Cross-Response Similarity (CRS) way applies RCScore dimensions to measure consistency across all possible combinations of instruction styles (6 pairwise comparisons).
  • ...and 3 more figures