Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Jellyfish: A Large Language Model for Data Preprocessing

Haochen Zhang, Yuyang Dong, Chuan Xiao, Masafumi Oyamada

TL;DR

This work addresses data preprocessing (DP) in data mining and the privacy concerns of API-based LLMs by introducing Jellyfish, a set of instruction-tuned local LLMs (7B, 8B, 13B) that solve universal DP tasks on a single GPU. By constructing DP task data and reasoning data, plus domain knowledge injection, Jellyfish achieves competitive DP performance against GPT-3.5/4 and demonstrates strong generalization to unseen tasks while preserving NLP capabilities. The study also shows interpretability advantages through reasoning prompts and analyzes the effects of knowledge injection, data configuration, and reasoning data. Practically, Jellyfish offers a privacy-preserving, customizable, and scalable approach to DP that can be extended to new tasks with minimal API exposure, making it valuable for real-world data pipelines.

Abstract

This paper explores the utilization of LLMs for data preprocessing (DP), a crucial step in the data mining pipeline that transforms raw data into a clean format conducive to easy processing. Whereas the use of LLMs has sparked interest in devising universal solutions to DP, recent initiatives in this domain typically rely on GPT APIs, raising inevitable data breach concerns. Unlike these approaches, we consider instruction-tuning local LLMs (7 -- 13B models) as universal DP task solvers that operate on a local, single, and low-priced GPU, ensuring data security and enabling further customization. We select a collection of datasets across four representative DP tasks and construct instruction tuning data using data configuration, knowledge injection, and reasoning data distillation techniques tailored to DP. By tuning Mistral-7B, Llama 3-8B, and OpenOrca-Platypus2-13B, our models, namely, Jellyfish-7B/8B/13B, deliver competitiveness compared to GPT-3.5/4 models and strong generalizability to unseen tasks while barely compromising the base models' abilities in NLP tasks. Meanwhile, Jellyfish offers enhanced reasoning capabilities compared to GPT-3.5. Our models are available at: https://huggingface.co/NECOUDBFM/Jellyfish . Our instruction dataset is available at: https://huggingface.co/datasets/NECOUDBFM/Jellyfish-Instruct .

Jellyfish: A Large Language Model for Data Preprocessing

TL;DR

This work addresses data preprocessing (DP) in data mining and the privacy concerns of API-based LLMs by introducing Jellyfish, a set of instruction-tuned local LLMs (7B, 8B, 13B) that solve universal DP tasks on a single GPU. By constructing DP task data and reasoning data, plus domain knowledge injection, Jellyfish achieves competitive DP performance against GPT-3.5/4 and demonstrates strong generalization to unseen tasks while preserving NLP capabilities. The study also shows interpretability advantages through reasoning prompts and analyzes the effects of knowledge injection, data configuration, and reasoning data. Practically, Jellyfish offers a privacy-preserving, customizable, and scalable approach to DP that can be extended to new tasks with minimal API exposure, making it valuable for real-world data pipelines.

Abstract

This paper explores the utilization of LLMs for data preprocessing (DP), a crucial step in the data mining pipeline that transforms raw data into a clean format conducive to easy processing. Whereas the use of LLMs has sparked interest in devising universal solutions to DP, recent initiatives in this domain typically rely on GPT APIs, raising inevitable data breach concerns. Unlike these approaches, we consider instruction-tuning local LLMs (7 -- 13B models) as universal DP task solvers that operate on a local, single, and low-priced GPU, ensuring data security and enabling further customization. We select a collection of datasets across four representative DP tasks and construct instruction tuning data using data configuration, knowledge injection, and reasoning data distillation techniques tailored to DP. By tuning Mistral-7B, Llama 3-8B, and OpenOrca-Platypus2-13B, our models, namely, Jellyfish-7B/8B/13B, deliver competitiveness compared to GPT-3.5/4 models and strong generalizability to unseen tasks while barely compromising the base models' abilities in NLP tasks. Meanwhile, Jellyfish offers enhanced reasoning capabilities compared to GPT-3.5. Our models are available at: https://huggingface.co/NECOUDBFM/Jellyfish . Our instruction dataset is available at: https://huggingface.co/datasets/NECOUDBFM/Jellyfish-Instruct .
Paper Structure (32 sections, 7 figures, 16 tables)

This paper contains 32 sections, 7 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Data Preprocessing
  4. Large Language Models
  5. Instruction Tuning of Jellyfish
  6. Dataset Preparation
  7. DP Task Data
  8. DP Task with Reasoning Data
  9. Inference with Jellyfish
  10. Extensions to Unseen Tasks
  11. Experiments
  12. Experimental Setup
  13. DP Performance
  14. Improvement of Jellyfish over Base Models
  15. Impact of Knowledge Injection
  16. ...and 17 more sections

Figures (7)

  • Figure 1: Overview of instruction tuning for data preprocessing.
  • Figure 2: Example prompt in instruction data. The leftmost column is description and not prompted to the model. Response indicates the answer to the prompt. Detailed prompts are provided in Appendix \ref{['sec:app:prompts']}.
  • Figure 3: Impact of tuning with single-task data on DP performance, zero-shot. Above red line is positive.
  • Figure 4: Impact of tuning with multi-task data on DP performance. Numbers in parenthesis indicate the percentage of data used for each task.
  • Figure 5: Impact of tuning with single-task data on NLP performance. Above red line is positive.
  • ...and 2 more figures