Unitxt: Flexible, Shareable and Reusable Data Preparation and Evaluation for Generative AI

TL;DR

Unitxt tackles the lack of standardization in textual data processing for LLMs by introducing a modular, recipe-based framework that unifies data preparation and evaluation across datasets, tasks, languages, and prompt structures. It decomposes pipelines into five ingredients—Resources, Task, Template, Format, Extensions—assembled into Recipes and managed in the Unitxt Catalog, enabling reuse and sharing. The library integrates with HuggingFace datasets and LM-eval-harness, allowing easy adoption without breaking existing workflows, and supports thousands of configurations (over 100K reported). Unitxt's community-driven design aims to accelerate reproducible experimentation and scalable training and evaluation of LLMs. The work demonstrates practical impact through IBM adoption and a growing public catalog.

Abstract

In the dynamic landscape of generative NLP, traditional text processing pipelines limit research flexibility and reproducibility, as they are tailored to specific dataset, task, and model combinations. The escalating complexity, involving system prompts, model-specific formats, instructions, and more, calls for a shift to a structured, modular, and customizable solution. Addressing this need, we present Unitxt, an innovative library for customizable textual data preparation and evaluation tailored to generative language models. Unitxt natively integrates with common libraries like HuggingFace and LM-eval-harness and deconstructs processing flows into modular components, enabling easy customization and sharing between practitioners. These components encompass model-specific formats, task prompts, and many other comprehensive dataset processing definitions. The Unitxt-Catalog centralizes these components, fostering collaboration and exploration in modern textual data workflows. Beyond being a tool, Unitxt is a community-driven platform, empowering users to build, share, and advance their pipelines collaboratively. Join the Unitxt community at https://github.com/IBM/unitxt!

Figures (4)

• Figure 1: Unitxt flow: The upper section illustrates the data-preparation pipeline §\ref{['data_preperation_pipeline']}, encompassing raw dataset loading, standardization according to the task interface, verbalization using templates, and application of formatting. The lower section showcases the evaluation pipeline §\ref{['evaluation_pipline']}, involving de-verbalization operations and output standardization before performance evaluation with task-defined metrics. All components are described in §\ref{['format']}.
• Figure 2: Illustration of the data preparation pipeline (§\ref{['data_preperation_pipeline']}), depicting the transformation from raw data and formatting specifications to the final text output. Components include Resources (raw data), Format (model-specific formatting requirements), and Template (verbalization).
• Figure 3: Exploration UI showcasing configuration options for model input creation on the left, including parameters such as task, dataset card, template, system-prompt, response-schema, number of examples, and optional augmentations. The resulting model input is displayed in the prompt window.
• Figure 4: Unitxt and LM-eval-harness integration. A Unitxt recipe can be integrated as an LM-eval-harness task, by setting the dataset_path (line 3) to unitxt/data and the setting the recipe in the dataset_name (line 4). Unitxt metrics can be used like any other metric (line 14).