Auto-BenchmarkCard: Automated Synthesis of Benchmark Documentation

TL;DR

This work tackles the problem of sparse and inconsistent AI benchmark documentation by introducing Auto-BenchmarkCard, a modular workflow that performs multi-agent data extraction from diverse sources (Unitxt, Hugging Face, academic papers), uses LLMs to synthesize a BenchmarkCard aligned with Sokol's schema, and validates factuality via FactReasoner with atomic entailment. The pipeline includes a vector-based retrieval of evidence and supports automated revision or human-in-the-loop corrections, while also incorporating a governance-oriented Risk Atlas Nexus. The approach yields a reproducible, open-source toolchain for generating validated benchmark documentation, enhancing transparency, comparability, and reusability across AI benchmarks and governance contexts.

Abstract

We present Auto-BenchmarkCard, a workflow for generating validated descriptions of AI benchmarks. Benchmark documentation is often incomplete or inconsistent, making it difficult to interpret and compare benchmarks across tasks or domains. Auto-BenchmarkCard addresses this gap by combining multi-agent data extraction from heterogeneous sources (e.g., Hugging Face, Unitxt, academic papers) with LLM-driven synthesis. A validation phase evaluates factual accuracy through atomic entailment scoring using the FactReasoner tool. This workflow has the potential to promote transparency, comparability, and reusability in AI benchmark reporting, enabling researchers and practitioners to better navigate and evaluate benchmark choices.

Figures (1)

• Figure 1: Auto-BenchmarkCard workflow overview