Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

WikiTableEdit: A Benchmark for Table Editing by Natural Language Instruction

Zheng Li, Xiang Chen, Xiaojun Wan

TL;DR

WikiTableEdit introduces a benchmark for direct natural-language editing of both regular and irregular tables, addressing challenges posed by merged cells via a 6-operation schema and HTML representations. It builds from WikiSQL to generate over 200k instruction–source–target triplets and proposes Table Edit Distance (TED) to quantify concurrent structural and content changes, including irregularities via rowspan matrices. Through zero-shot, few-shot, and supervised fine-tuning experiments across multiple LLMs, the study shows that current models plateau on this task, with GPT-3.5-turbo performing best in zero-shot but still far from perfect, and substantial gains achieved after fine-tuning. The work demonstrates the task's difficulty, especially for swap/reorder operations and irregular-table edits, and positions WikiTableEdit as a key resource for advancing NL-guided table editing and evaluation.

Abstract

Tabular data, as a crucial form of data representation, exists in diverse formats on the Web. When confronted with complex and irregular tables, manual modification becomes a laborious task. This paper investigates the performance of Large Language Models (LLMs) in the context of table editing tasks. Existing research mainly focuses on regular-shaped tables, wherein instructions are used to generate code in SQL, Python, or Excel Office-script for manipulating the tables. Nevertheless, editing tables with irregular structures, particularly those containing merged cells spanning multiple rows, poses a challenge when using code. To address this, we introduce the WikiTableEdit dataset. Leveraging 26,531 tables from the WikiSQL dataset, we automatically generate natural language instructions for six distinct basic operations and the corresponding outcomes, resulting in over 200,000 instances. Subsequently, we evaluate several representative large language models on the WikiTableEdit dataset to demonstrate the challenge of this task. The dataset will be released to the community to promote related researches.

WikiTableEdit: A Benchmark for Table Editing by Natural Language Instruction

TL;DR

WikiTableEdit introduces a benchmark for direct natural-language editing of both regular and irregular tables, addressing challenges posed by merged cells via a 6-operation schema and HTML representations. It builds from WikiSQL to generate over 200k instruction–source–target triplets and proposes Table Edit Distance (TED) to quantify concurrent structural and content changes, including irregularities via rowspan matrices. Through zero-shot, few-shot, and supervised fine-tuning experiments across multiple LLMs, the study shows that current models plateau on this task, with GPT-3.5-turbo performing best in zero-shot but still far from perfect, and substantial gains achieved after fine-tuning. The work demonstrates the task's difficulty, especially for swap/reorder operations and irregular-table edits, and positions WikiTableEdit as a key resource for advancing NL-guided table editing and evaluation.

Abstract

Tabular data, as a crucial form of data representation, exists in diverse formats on the Web. When confronted with complex and irregular tables, manual modification becomes a laborious task. This paper investigates the performance of Large Language Models (LLMs) in the context of table editing tasks. Existing research mainly focuses on regular-shaped tables, wherein instructions are used to generate code in SQL, Python, or Excel Office-script for manipulating the tables. Nevertheless, editing tables with irregular structures, particularly those containing merged cells spanning multiple rows, poses a challenge when using code. To address this, we introduce the WikiTableEdit dataset. Leveraging 26,531 tables from the WikiSQL dataset, we automatically generate natural language instructions for six distinct basic operations and the corresponding outcomes, resulting in over 200,000 instances. Subsequently, we evaluate several representative large language models on the WikiTableEdit dataset to demonstrate the challenge of this task. The dataset will be released to the community to promote related researches.
Paper Structure (18 sections, 3 figures, 5 tables)

This paper contains 18 sections, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Task Overview
  3. Dataset Construction
  4. Automatic Generation
  5. Dataset Analysis
  6. Examples of WikiTableEdit
  7. Experiments
  8. Evaluation Metrics
  9. Experiment Details
  10. Results
  11. Different Editing Operations
  12. Regular vs. Irregular Table Editing
  13. Manually Check
  14. Related Works
  15. Conclusion
  16. ...and 3 more sections

Figures (3)

  • Figure 1: Illustration for the table editing task.
  • Figure 2: An example for rowspan matrix extraction.
  • Figure 3: Examples for WikiTableEdit