ModelTables: A Corpus of Tables about Models

TL;DR

ModelTables introduces a large-scale benchmark of tables describing AI models, linking each table to model cards, data cards, code repositories, and scholarly papers to enable structured semantic retrieval beyond text. It constructs multi-source ground-truth relatedness from paper citations, model-card lineage, and shared datasets, and provides an end-to-end extraction and augmentation pipeline from Hugging Face Model Lake, GitHub READMEs, arXiv, and Semantic Scholar. The paper demonstrates a comprehensive evaluation of table search methods (keyword, joinable, union, dense, sparse, and hybrid) against the ground-truth, revealing substantial room for improvement and showing that semantic and structural augmentations can improve retrieval robustness. By releasing ModelTables and its creation protocol, the work offers a first large-scale, reproducible benchmark for structured model knowledge and illustrates how table discovery can enhance model understanding and governance in model lakes.

Abstract

We present ModelTables, a benchmark of tables in Model Lakes that captures the structured semantics of performance and configuration tables often overlooked by text only retrieval. The corpus is built from Hugging Face model cards, GitHub READMEs, and referenced papers, linking each table to its surrounding model and publication context. Compared with open data lake tables, model tables are smaller yet exhibit denser inter table relationships, reflecting tightly coupled model and benchmark evolution. The current release covers over 60K models and 90K tables. To evaluate model and table relatedness, we construct a multi source ground truth using three complementary signals: (1) paper citation links, (2) explicit model card links and inheritance, and (3) shared training datasets. We present one extensive empirical use case for the benchmark which is table search. We compare canonical Data Lake search operators (unionable, joinable, keyword) and Information Retrieval baselines (dense, sparse, hybrid retrieval) on this benchmark. Union based semantic table retrieval attains 54.8 % P@1 overall (54.6 % on citation, 31.3 % on inheritance, 30.6 % on shared dataset signals); table based dense retrieval reaches 66.5 % P@1, and metadata hybrid retrieval achieves 54.1 %. This evaluation indicates clear room for developing better table search methods. By releasing ModelTables and its creation protocol, we provide the first large scale benchmark of structured data describing AI model. Our use case of table discovery in Model Lakes, provides intuition and evidence for developing more accurate semantic retrieval, structured comparison, and principled organization of structured model knowledge. Source code, data, and other artifacts have been made available at https://github.com/RJMillerLab/ModelTables.

Figures (8)

• Figure 1: The Model Lake Benchmark Setup Pipeline. This figure outlines our end-to-end, automated workflow for establishing the Model Lake benchmark. It illustrates the general-purpose pipeline for collecting, cleaning, and linking diverse tables, including the integration of a paper-level citation graph. Furthermore, it details how we construct a multi-level ground-truth for model relatedness by combining signals from paper citations, model lineage from model cards, and shared dataset metadata, providing a robust basis for evaluation.
• Figure 2: Tables from RoBERTa liu2019roberta and DPS zhang2022fine motivate augmentations: transposition for alignment, and header–cell fusion for semantic normalization
• Figure 3: Model- and table-level filtering of the model lake. Top: Stepwise filtering of models by card completeness, table presence, and valid references, yielding the final benchmark subset. Bottom: Table frequency distribution showing a long-tailed reuse pattern that motivates deduplication.
• Figure 4: Our benchmark contains a larger quantity of smaller tables compared to data lakes, a structural profile driven by the distinct characteristics of its academic sources.
• Figure 5: The pronounced long-tail distribution in our ground truth arises from influential models acting as hubs, which link to many model cards.