TableLlama: Towards Open Large Generalist Models for Tables

TL;DR

This paper introduces TableInstruct, a large-scale instruction-tuning benchmark for open generalist table models, and TableLlama, an open-source 7B LLM fine-tuned with LongLoRA to handle long-context table tasks. The dataset blends 14 real-world table datasets across 11 tasks, spanning in-domain and out-of-domain settings, and emphasizes realistic tasks without simplifying assumptions. Empirical results show TableLlama achieves competitive or superior performance to task-specific SOTA on most in-domain tasks and gains 5–44 absolute points on several out-of-domain datasets, illustrating strong generalization from comprehensive instruction tuning. The work advocates for open, end-to-end generalist capabilities for tables and provides resources to advance research in open-table AI systems.

Abstract

Semi-structured tables are ubiquitous. There has been a variety of tasks that aim to automatically interpret, augment, and query tables. Current methods often require pretraining on tables or special model architecture design, are restricted to specific table types, or have simplifying assumptions about tables and tasks. This paper makes the first step towards developing open-source large language models (LLMs) as generalists for a diversity of table-based tasks. Towards that end, we construct TableInstruct, a new dataset with a variety of realistic tables and tasks, for instruction tuning and evaluating LLMs. We further develop the first open-source generalist model for tables, TableLlama, by fine-tuning Llama 2 (7B) with LongLoRA to address the long context challenge. We experiment under both in-domain setting and out-of-domain setting. On 7 out of 8 in-domain tasks, TableLlama achieves comparable or better performance than the SOTA for each task, despite the latter often has task-specific design. On 6 out-of-domain datasets, it achieves 5-44 absolute point gains compared with the base model, showing that training on TableInstruct enhances the model's generalizability. We open-source our dataset and trained model to boost future work on developing open generalist models for tables.

Figures (16)

• Figure 1: An overview of TableInstruct and TableLlama. TableInstruct includes a wide variety of realistic tables and tasks with instructions. We make the first step towards developing open-source generalist models for tables with TableInstruct and TableLlama.
• Figure 2: Illustration of three exemplary tasks: (a) Column type annotation. This task is to annotate the selected column with the correct semantic types. (b) Row population. This task is to populate rows given table metadata and partial row entities. (c) Hierarchical table QA. For subfigures (a) and (b), we mark candidates with red color in the "task instruction" part. The candidate set size can be hundreds to thousands in TableInstruct.
• Figure 3: Column type annotation task. This task is to annotate the selected column with the correct semantic types. We mark candidates with red color in the "task instruction" part. The candidate size can be up to hundreds to thousands in TableInstruct.
• Figure 4: Relation extraction task. This task is to annotate the selected column pairs with the correct relations. We mark candidates with red color in the "task instruction" part. The candidate size can be up to hundreds to thousands in TableInstruct.
• Figure 5: Entity linking task. This task is to link the selected entity mention in the table cells to the entity in the knowledge base. We mark candidates with red color in the "task instruction" part. The candidate size can be up to hundreds to thousands in TableInstruct.