StructLM: Towards Building Generalist Models for Structured Knowledge Grounding

TL;DR

This investigation reveals a notable deficiency in LLMs' ability to process structured data, e.g., ChatGPT lags behind state-of-the-art (SoTA) model by an average of 35% and suggests that structured knowledge grounding is still a challenging task and requires more innovative design to push to a new level.

Abstract

Structured data sources, such as tables, graphs, and databases, are ubiquitous knowledge sources. Despite the demonstrated capabilities of large language models (LLMs) on plain text, their proficiency in interpreting and utilizing structured data remains limited. Our investigation reveals a notable deficiency in LLMs' ability to process structured data, e.g., ChatGPT lags behind state-of-the-art (SoTA) model by an average of 35%. To augment the Structured Knowledge Grounding (SKG) capabilities in LLMs, we have developed a comprehensive instruction tuning dataset comprising 1.1 million examples. Utilizing this dataset, we train a series of models, referred to as StructLM, based on the Mistral and the CodeLlama model family, ranging from 7B to 34B parameters. Our StructLM series surpasses task-specific models on 16 out of 18 evaluated datasets and establishes new SoTA performance on 8 SKG tasks. Furthermore, StructLM demonstrates strong generalization across 6 novel held-out SKG tasks, outperforming TableLlama by an average of 35\% and Flan-UL2 20B by an average of 10\%. Contrary to expectations, we observe that scaling model size offers marginal benefits, with StructLM-34B showing only slight improvements over StructLM-7B. This suggests that structured knowledge grounding is still a challenging task and requires more innovative design to push to a new level.

Figures (6)

• Figure 1: StructLM can ground on structured and unstructured knowledge to respond to human queries. The previous SoTA was attained by many different task-specific models like TAPEX liu2021tapex, USKG UnifiedSKG2022, TableLlama zhang_tablellama_2023, BINDER-Codex cheng2022binding, etc. StructLM beats the SoTAs on seven SKG tasks.
• Figure 2: Overview of StructLM. This figure illustrates the prompting structure of StructLM, highlighting its capability to process various forms of structured data beyond linearized data tables, including linearized database schemas and knowledge graphs.
• Figure 3: Breakdown of Structured Knowledge Types and Tasks in the Training Data. On the left side, we see a coarse breakdown of the different categories of structured inputs in our dataset. On the right side, we see an overview of the task groups that are represented for those structured knowledge types.
• Figure 4: Effect of different pretraining curricula on SKG finetuning performance in relevant task groupings. We can observe the advantages of CodeLlma over the others.
• Figure 5: Effect of general instruction-following data on averaged held-out SKG dataset performance. Performance is measured as the average over evaluation metrics across all tasks within held-in or held-out groups.