Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Better Understanding Table Instruction Tuning: Decoupling the Effects from Data versus Models

Naihao Deng, Sheng Zhang, Henghui Zhu, Shuaichen Chang, Jiani Zhang, Alexander Hanbo Li, Chung-Wei Hang, Hideo Kobayashi, Yiqun Hu, Patrick Ng

TL;DR

This work fine-tune base models from the Mistral, OLMo, and Phi families on existing public training datasets, and achieves performance on par with or surpassing existing table LLMs, establishing new state-of-the-art performance on Hitab, a table question-answering dataset.

Abstract

Recent advances in natural language processing have leveraged instruction tuning to enhance Large Language Models (LLMs) for table-related tasks. However, previous works train different base models with different training data, lacking an apples-to-apples comparison across the result table LLMs. To address this, we fine-tune base models from the Mistral, OLMo, and Phi families on existing public training datasets. Our replication achieves performance on par with or surpassing existing table LLMs, establishing new state-of-the-art performance on Hitab, a table question-answering dataset. More importantly, through systematic out-of-domain evaluation, we decouple the contributions of training data and the base model, providing insight into their individual impacts. In addition, we assess the effects of table-specific instruction tuning on general-purpose benchmarks, revealing trade-offs between specialization and generalization.

Towards Better Understanding Table Instruction Tuning: Decoupling the Effects from Data versus Models

TL;DR

This work fine-tune base models from the Mistral, OLMo, and Phi families on existing public training datasets, and achieves performance on par with or surpassing existing table LLMs, establishing new state-of-the-art performance on Hitab, a table question-answering dataset.

Abstract

Recent advances in natural language processing have leveraged instruction tuning to enhance Large Language Models (LLMs) for table-related tasks. However, previous works train different base models with different training data, lacking an apples-to-apples comparison across the result table LLMs. To address this, we fine-tune base models from the Mistral, OLMo, and Phi families on existing public training datasets. Our replication achieves performance on par with or surpassing existing table LLMs, establishing new state-of-the-art performance on Hitab, a table question-answering dataset. More importantly, through systematic out-of-domain evaluation, we decouple the contributions of training data and the base model, providing insight into their individual impacts. In addition, we assess the effects of table-specific instruction tuning on general-purpose benchmarks, revealing trade-offs between specialization and generalization.
Paper Structure (57 sections, 4 figures, 11 tables)

This paper contains 57 sections, 4 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Table-Related Tasks.
  4. Methods for Table Understanding.
  5. Replicating Existing Table LLMs
  6. Issues with Comparing Existing Table LLMs
  7. Foundational LLM Selections.
  8. Experimental Setups.
  9. Comparison of Our Models v.s. the Original Models
  10. Replicating TableLlama
  11. Training Datasets.
  12. Evaluation Datasets.
  13. Comparison.
  14. Replicating TableLLM
  15. Training Datasets.
  16. ...and 42 more sections

Figures (4)

  • Figure 1: Performance of fine-tuned models trained on different data (e.g. TableLlama) on general benchmarks. The green and red hatched bars represent performance gains or losses relative to the base model, respectively. On IFEval, unlike other models, the Mistral model shows a significant performance drop, underscoring the impact of innate model capabilities on preserving general performance after domain-specific fine-tuning.
  • Figure 2: Performance of Phi 3 Mini Instruct (4B) versus Phi 3 Small Instruct (7B) model on different table tasks with different training data. In most cases, the 7B model outperforms the 4B model.
  • Figure 3: Performance of Phi 3 Mini Instruct (4B) versus Phi 3 Small Instruct (7B) model on different table tasks with different training data. In most cases, the 7B model outperforms the 4B model.
  • Figure 4: Performance difference between Phi 3 Mini Instruct (4B) versus Phi 3 Small Instruct (7B) model. On MMLU, MMLUPro, IFEval, the Small (7B) version yields better performance both before and after fine-tuning. On GPQA, the two models perform comparably. On AI2ARC, the Mini (4B) version yields better performance.