Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data

Pengcheng Yin, Graham Neubig, Wen-tau Yih, Sebastian Riedel

TL;DR

TaBert introduces a pretraining regime and encoder design for joint understanding of natural language and tabular data, addressing semantic parsing over tables. By using content snapshots, row-level transformers, and vertical self-attention, TaBert learns representations for utterances and table schemas from a large NL-table corpus. It achieves state-of-the-art results on WikiTableQuestions and competitive performance on Spider, demonstrating the value of aligned NL-table pretraining for downstream semantic parsing. The work offers a general, table-aware encoder that can enhance parsing across both structured and semi-structured data domains.

Abstract

Recent years have witnessed the burgeoning of pretrained language models (LMs) for text-based natural language (NL) understanding tasks. Such models are typically trained on free-form NL text, hence may not be suitable for tasks like semantic parsing over structured data, which require reasoning over both free-form NL questions and structured tabular data (e.g., database tables). In this paper we present TaBERT, a pretrained LM that jointly learns representations for NL sentences and (semi-)structured tables. TaBERT is trained on a large corpus of 26 million tables and their English contexts. In experiments, neural semantic parsers using TaBERT as feature representation layers achieve new best results on the challenging weakly-supervised semantic parsing benchmark WikiTableQuestions, while performing competitively on the text-to-SQL dataset Spider. Implementation of the model will be available at http://fburl.com/TaBERT .

TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data

TL;DR

TaBert introduces a pretraining regime and encoder design for joint understanding of natural language and tabular data, addressing semantic parsing over tables. By using content snapshots, row-level transformers, and vertical self-attention, TaBert learns representations for utterances and table schemas from a large NL-table corpus. It achieves state-of-the-art results on WikiTableQuestions and competitive performance on Spider, demonstrating the value of aligned NL-table pretraining for downstream semantic parsing. The work offers a general, table-aware encoder that can enhance parsing across both structured and semi-structured data domains.

Abstract

Recent years have witnessed the burgeoning of pretrained language models (LMs) for text-based natural language (NL) understanding tasks. Such models are typically trained on free-form NL text, hence may not be suitable for tasks like semantic parsing over structured data, which require reasoning over both free-form NL questions and structured tabular data (e.g., database tables). In this paper we present TaBERT, a pretrained LM that jointly learns representations for NL sentences and (semi-)structured tables. TaBERT is trained on a large corpus of 26 million tables and their English contexts. In experiments, neural semantic parsers using TaBERT as feature representation layers achieve new best results on the challenging weakly-supervised semantic parsing benchmark WikiTableQuestions, while performing competitively on the text-to-SQL dataset Spider. Implementation of the model will be available at http://fburl.com/TaBERT .

Paper Structure

This paper contains 45 sections, 6 equations, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Semantic Parsing over Tables
  4. Masked Language Models
  5. TaBert: Learning Joint Representa- tions over Textual and Tabular Data
  6. Computing Representations for NL Utterances and Table Schemas
  7. Content Snapshot
  8. Row Linearization
  9. Vertical Self-Attention Mechanism
  10. Utterance and Column Representations
  11. Pretraining Procedure
  12. Training Data
  13. Unsupervised Learning Objectives
  14. Example Application: Semantic Parsing over Tables
  15. Supervised Semantic Parsing
  16. ...and 30 more sections

Figures (2)

  • Figure 1: Overview of TaBert for learning representations of utterances and table schemas with an example from WikiTableQuestions. (A) A content snapshot of the table is created based on the input NL utterance. (B) Each row in the snapshot is encoded by a Transformer (only $R_2$ is shown), producing row-wise encodings for utterance tokens and cells. (C) All row-wise encodings are aligned and processed by $V$ vertical self-attention layers, generating utterance and column representations.
  • Figure 2: ASDL Grammar of SemQL used in TranX