Metric Calculating Benchmark: Code-Verifiable Complicate Instruction Following Benchmark for Large Language Models

TL;DR

MCBench tackles the need for tougher, objective benchmarks beyond saturated instruction‑following tasks by introducing a step‑by‑step rubric to compute a string‑matching metric, validated with a parallel reference code. It evaluates three capabilities—complex instruction following, mathematical reasoning, and long‑range consistency—across multiple input modalities using three metrics: $FA$, $FF$, and $FD$. The study finds that even state‑of‑the‑art models struggle to fully comply with the rubric and that focused specializations offer limited gains, underscoring the requirement for holistic improvement. The framework provides a publicly available, code‑verifiable benchmark that enables objective comparison of advanced LLMs and sets the stage for future tool‑driven evaluation approaches.

Abstract

Recent frontier-level LLMs have saturated many previously difficult benchmarks, leaving little room for further differentiation. This progress highlights the need for challenging benchmarks that provide objective verification. In this paper, we introduce MCBench, a benchmark designed to evaluate whether LLMs can execute string-matching NLP metrics by strictly following step-by-step instructions. Unlike prior benchmarks that depend on subjective judgments or general reasoning, MCBench offers an objective, deterministic and codeverifiable evaluation. This setup allows us to systematically test whether LLMs can maintain accurate step-by-step execution, including instruction adherence, numerical computation, and long-range consistency in handling intermediate results. To ensure objective evaluation of these abilities, we provide a parallel reference code that can evaluate the accuracy of LLM output. We provide three evaluative metrics and three benchmark variants designed to measure the detailed instruction understanding capability of LLMs. Our analyses show that MCBench serves as an effective and objective tool for evaluating the capabilities of cutting-edge LLMs.

Figures (4)

• Figure 1: MCBench comprises a metric set and a candidate set. Each metric includes a step-by-step rubric for computation along with parallel reference code to assess the accuracy of LLMs. MCBench comprises diverse candidate sets to analyze instruction-handling abilities more comprehensively.
• Figure 2: Overall Data Construction Process. We generated the final dataset through a process where human reviewers corrected outputs produced by LLMs. Each data point was finalized only after thorough human verification.
• Figure 3: Final Accuracy (FA) score for each metric
• Figure 4: Performance differences based on the level of rubric details, which are reported using the FA metric