MiMoTable: A Multi-scale Spreadsheet Benchmark with Meta Operations for Table Reasoning

TL;DR

MiMoTable introduces a multi-scale spreadsheet benchmark that uses real-world Excel files across seven domains and injects a meta-operations framework to quantify question difficulty. It spans four tasks—TableQA, Table2Text, Table Manipulation, and Advanced Data Analysis—and provides 1,719 QA triplets derived from 428 spreadsheets. The benchmark enables fine-grained difficulty analysis through six meta-operations and demonstrates that state-of-the-art LLMs, including Claude-3.5-Sonnet, still struggle, achieving 77.4% overall accuracy. The work highlights the value of meta-operations for cross-benchmark evaluation and motivates further advances in table reasoning, especially for complex headers, multiple sheets, and cross-file analyses.

Abstract

Extensive research has been conducted to explore the capability of Large Language Models (LLMs) for table reasoning and has significantly improved the performance on existing benchmarks. However, tables and user questions in real-world applications are more complex and diverse, presenting an unignorable gap compared to the existing benchmarks. To fill the gap, we propose a \textbf{M}ult\textbf{i}-scale spreadsheet benchmark with \textbf{M}eta \textbf{o}perations for \textbf{Table} reasoning, named as MiMoTable. Specifically, MiMoTable incorporates two key features. First, the tables in MiMoTable are all spreadsheets used in real-world scenarios, which cover seven domains and contain different types. Second, we define a new criterion with six categories of meta operations for measuring the difficulty of each question in MiMoTable, simultaneously as a new perspective for measuring the difficulty of the existing benchmarks. Experimental results show that Claude-3.5-Sonnet achieves the best performance with 77.4\% accuracy, indicating that there is still significant room to improve for LLMs on MiMoTable. Furthermore, we grade the difficulty of existing benchmarks according to our new criteria. Experiments have shown that the performance of LLMs decreases as the difficulty of benchmarks increases, thereby proving the effectiveness of our proposed new criterion.

Figures (10)

• Figure 1: Examples of MiMoTable benchmark.
• Figure 2: Illustrations of different table types, including simple header, complex header, single sheet, multiple sheets, multiple files, and multiple tables in one sheet.
• Figure 3: The relationships between tasks in the existing benchmarks and our proposed meta operations.
• Figure 4: The data construction pipeline of MiMoTable benchmark.
• Figure 5: Domain distribution of all spreadsheets.