Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can Foundation Models Wrangle Your Data?

Avanika Narayan, Ines Chami, Laurel Orr, Simran Arora, Christopher Ré

TL;DR

This work investigates whether large foundation models can tackle classical data tasks such as cleaning and integration by reframing them as natural language prompting problems. Using GPT-3-175B, the authors demonstrate zero- and few-shot transfer to tasks like entity matching, error detection, data transformation, and imputation, achieving state-of-the-art performance with minimal or no task-specific finetuning. They analyze prompt design, including tabular serialization, task prompts, and demonstration selection, showing that prompts and high-quality demonstrations are crucial for performance and stability. The study highlights practical opportunities and challenges for FM-enabled data management, including domain specialization, privacy, and the need for iterative, human-in-the-loop prompt engineering to realize scalable, accessible data pipelines.

Abstract

Foundation Models (FMs) are models trained on large corpora of data that, at very large scale, can generalize to new tasks without any task-specific finetuning. As these models continue to grow in size, innovations continue to push the boundaries of what these models can do on language and image tasks. This paper aims to understand an underexplored area of FMs: classical data tasks like cleaning and integration. As a proof-of-concept, we cast five data cleaning and integration tasks as prompting tasks and evaluate the performance of FMs on these tasks. We find that large FMs generalize and achieve SoTA performance on data cleaning and integration tasks, even though they are not trained for these data tasks. We identify specific research challenges and opportunities that these models present, including challenges with private and domain specific data, and opportunities to make data management systems more accessible to non-experts. We make our code and experiments publicly available at: https://github.com/HazyResearch/fm_data_tasks.

Can Foundation Models Wrangle Your Data?

TL;DR

This work investigates whether large foundation models can tackle classical data tasks such as cleaning and integration by reframing them as natural language prompting problems. Using GPT-3-175B, the authors demonstrate zero- and few-shot transfer to tasks like entity matching, error detection, data transformation, and imputation, achieving state-of-the-art performance with minimal or no task-specific finetuning. They analyze prompt design, including tabular serialization, task prompts, and demonstration selection, showing that prompts and high-quality demonstrations are crucial for performance and stability. The study highlights practical opportunities and challenges for FM-enabled data management, including domain specialization, privacy, and the need for iterative, human-in-the-loop prompt engineering to realize scalable, accessible data pipelines.

Abstract

Foundation Models (FMs) are models trained on large corpora of data that, at very large scale, can generalize to new tasks without any task-specific finetuning. As these models continue to grow in size, innovations continue to push the boundaries of what these models can do on language and image tasks. This paper aims to understand an underexplored area of FMs: classical data tasks like cleaning and integration. As a proof-of-concept, we cast five data cleaning and integration tasks as prompting tasks and evaluate the performance of FMs on these tasks. We find that large FMs generalize and achieve SoTA performance on data cleaning and integration tasks, even though they are not trained for these data tasks. We identify specific research challenges and opportunities that these models present, including challenges with private and domain specific data, and opportunities to make data management systems more accessible to non-experts. We make our code and experiments publicly available at: https://github.com/HazyResearch/fm_data_tasks.
Paper Structure (31 sections, 4 equations, 5 figures, 6 tables)

This paper contains 31 sections, 4 equations, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Problem Setup
  4. Background on Foundation Models
  5. Foundation Models for Data Tasks
  6. Tabular Data Serialization
  7. Data Tasks as Natural Language Tasks
  8. Task Demonstrations
  9. Experiments
  10. Experimental Setup
  11. Zero/Few-shot Performance of Large FMs
  12. Results
  13. Discussion
  14. Prompt Tuning Ablations
  15. Results
  16. ...and 16 more sections

Figures (5)

  • Figure 1: A large FM can address an entity matching task using prompting. Rows are serialized into text and passed to the FM with the question "Are products A and B the same?". The FM then generates a string "Yes" or "No" as the answer.
  • Figure 2: Different ways to use FMs with "in-context" learning brown2020language on an error detection task. For zero-shot (left), the prompt is the task description and the example to complete. For few-shot (right), the prompt adds demonstrations of how to complete the task.
  • Figure 3: Adapter architecture
  • Figure 4: Sample-training efficiency tradeoffs. Larger FMs can be directly used in a zero/few-shot fashion. To achieve similar performance, smaller FMs need additional finetuning.
  • Figure 5: Finetuning experiments: Smaller FMs can be finetuned to bridge the performance gap with larger FMs. Full-finetuing bridges this gap with less data than adapters, but adapters are significantly less expensive to train.