TAPEX: Table Pre-training via Learning a Neural SQL Executor

TL;DR

TaPEx introduces an execution-centric table pre-training paradigm that trains a neural SQL executor by exposing a language model to a synthetic corpus of SQL queries and their execution results on tables. Using a BART-based encoder–decoder, it then fine-tunes on downstream TableQA and TableFV tasks as sequence generation problems, achieving state-of-the-art results across WikiSQL-Weak, WikiTableQuestions, SQA, and TabFact. Key contributions include a scalable, high-quality synthetic pre-training corpus drawn from public tables and SQL templates, an explicit neural execution objective, and demonstrated efficiency and generalization in low-resource settings. The work highlights the potential of pre-training with executable programs to advance structured data understanding and reasoning in NLP models.

Abstract

Recent progress in language model pre-training has achieved a great success via leveraging large-scale unstructured textual data. However, it is still a challenge to apply pre-training on structured tabular data due to the absence of large-scale high-quality tabular data. In this paper, we propose TAPEX to show that table pre-training can be achieved by learning a neural SQL executor over a synthetic corpus, which is obtained by automatically synthesizing executable SQL queries and their execution outputs. TAPEX addresses the data scarcity challenge via guiding the language model to mimic a SQL executor on the diverse, large-scale and high-quality synthetic corpus. We evaluate TAPEX on four benchmark datasets. Experimental results demonstrate that TAPEX outperforms previous table pre-training approaches by a large margin and achieves new state-of-the-art results on all of them. This includes the improvements on the weakly-supervised WikiSQL denotation accuracy to 89.5% (+2.3%), the WikiTableQuestions denotation accuracy to 57.5% (+4.8%), the SQA denotation accuracy to 74.5% (+3.5%), and the TabFact accuracy to 84.2% (+3.2%). To our knowledge, this is the first work to exploit table pre-training via synthetic executable programs and to achieve new state-of-the-art results on various downstream tasks. Our code can be found at https://github.com/microsoft/Table-Pretraining.

Figures (9)

• Figure 1: The schematic overview of our method. For the sake of brevity, the table content in the input is simplified with the symbol $\mathtt{[Table]}$.
• Figure 2: The illustration of the fine-tuning procedure in our method. During fine-tuning, we feed the concatenation of an NL sentence and its corresponding table taken from the downstream task to the model, and train it to output the answer (e.g., "Marisela Moreno Montero").
• Figure 3: The illustration of the pre-training procedure in our method. During pre-training, we feed the concatenation of a sampled SQL query and a sampled table to the model, and train it to output the corresponding execution result (e.g., "Pairs").
• Figure 4: The visualization results of attention weights from other tokens to the cell "adrian lewis". Intuitively, the darker the color, the more closely the word is associated with "adrian lewis".
• Figure 5: The illustration of downstream tasks performance with different scales of pre-training corpus. Scaling up the pre-training corpus of TaPEx generally brings positive effects across datasets.