On Meta-Evaluation

TL;DR

This work formalizes meta-evaluation by introducing the evaluation-condition (EC) space and AxiaBench, a large-scale benchmark to compare ten evaluation methods across eight scientific domains. It reveals that no single method delivers both high accuracy and low cost across all tasks, with observational designs often biased and certain DoE variants unreliable under strict criteria; however, the evaluatology-based entire-space stratified sampling consistently offers the best accuracy-cost balance. The study provides ground-truth-based performance metrics, demonstrates cross-domain generalizability, and positions meta-evaluation as a practical, open framework for building trustworthy evaluation in computational and experimental research. By openly releasing tools and data, the authors advocate a broader scientific project to systematize evaluation itself as a universal, rigorous science of evidence.

Abstract

Evaluation is the foundation of empirical science, yet the evaluation of evaluation itself -- so-called meta-evaluation -- remains strikingly underdeveloped. While methods such as observational studies, design of experiments (DoE), and randomized controlled trials (RCTs) have shaped modern scientific practice, there has been little systematic inquiry into their comparative validity and utility across domains. Here we introduce a formal framework for meta-evaluation by defining the evaluation space, its structured representation, and a benchmark we call AxiaBench. AxiaBench enables the first large-scale, quantitative comparison of ten widely used evaluation methods across eight representative application domains. Our analysis reveals a fundamental limitation: no existing method simultaneously achieves accuracy and efficiency across diverse scenarios, with DoE and observational designs in particular showing significant deviations from real-world ground truth. We further evaluate a unified method of entire-space stratified sampling from previous evaluatology research, and the results report that it consistently outperforms prior approaches across all tested domains. These results establish meta-evaluation as a scientific object in its own right and provide both a conceptual foundation and a pragmatic tool set for advancing trustworthy evaluation in computational and experimental research.

Figures (6)

• Figure 1: The complete framework of meta-evaluation involves the assessment of a specific evaluation methodology, shown using Large Language Model evaluation task as the example. In meta-evaluation, the EC space is that of the original EC along with the original evaluation object, while the meta-evaluation object is a certain evaluation method. The ground truth is established as the mean outcome across the entire spectrum (i.e. six dimensions in the example). Subsequently, the performance of the methodology is juxtaposed with the ground truth, reflecting both its precision and efficiency.
• Figure 2: The accuracy–cost curves for various evaluation methods in Tasks 1, 2 and 3 are shown above. Colors represent different methods, shapes correspond to objects, and line styles denote evaluation metrics. The red lines indicate the Evaluatology method, which achieves the highest performance.
• Figure 3: The accuracy–cost curves for various evaluation methods in Tasks 4, 5 and 6 are shown above. Colors represent different methods, shapes correspond to objects, and line styles denote evaluation metrics. The red lines indicate the Evaluatology method, which achieves the highest performance.
• Figure 4: The accuracy–cost curves for various evaluation methods in Tasks 7 and 8 are shown above. Colors represent different methods, shapes correspond to objects, and line styles denote evaluation metrics. The red lines indicate the Evaluatology method, which achieves the highest performance.
• Figure 5: The process and structure of meta-evaluation and its implementation on assessment methodologies, encompassing our innovative evaluatology approach.