Eval Factsheets: A Structured Framework for Documenting AI Evaluations

TL;DR

The paper identifies a critical gap in documenting AI evaluation methodologies and introduces Eval Factsheets, a five-dimensional taxonomy (Context, Scope, Structure, Method, Alignment) implemented as a practical, questionnaire-based framework. Through case studies on ImageNet, HumanEval, and MT-Bench, it demonstrates the framework's ability to capture diverse evaluation paradigms, including LLM-as-judge approaches, while enabling comparability and reproducibility. It also provides integration guidance with existing documentation ecosystems and concrete design principles to balance comprehensiveness with usability. The work aims to standardize evaluation reporting to improve transparency, enable meta-analysis, and facilitate adoption across research and deployment contexts.

Abstract

The rapid proliferation of benchmarks has created significant challenges in reproducibility, transparency, and informed decision-making. However, unlike datasets and models -- which benefit from structured documentation frameworks like Datasheets and Model Cards -- evaluation methodologies lack systematic documentation standards. We introduce Eval Factsheets, a structured, descriptive framework for documenting AI system evaluations through a comprehensive taxonomy and questionnaire-based approach. Our framework organizes evaluation characteristics across five fundamental dimensions: Context (Who made the evaluation and when?), Scope (What does it evaluate?), Structure (With what the evaluation is built?), Method (How does it work?) and Alignment (In what ways is it reliable/valid/robust?). We implement this taxonomy as a practical questionnaire spanning five sections with mandatory and recommended documentation elements. Through case studies on multiple benchmarks, we demonstrate that Eval Factsheets effectively captures diverse evaluation paradigms -- from traditional benchmarks to LLM-as-judge methodologies -- while maintaining consistency and comparability. We hope Eval Factsheets are incorporated into both existing and newly released evaluation frameworks and lead to more transparency and reproducibility.