The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards

TL;DR

The paper addresses the problem of opaque data quality in AI systems by introducing the Dataset Nutrition Label, a modular framework that combines qualitative and quantitative modules with probabilistic backends to interrogate datasets before model development. It details a web-based ecosystem with a label maker and label viewer that produce standardized, interactive labels, demonstrated via an open-source prototype on the ProPublica Dollars for Docs dataset using seven modules including Metadata, Variables, Statistics, Pair Plots, Probabilistic Modeling, and Ground Truth Correlations. The work shows how such labels can accelerate data interrogation, improve dataset selection, and potentially enhance AI accountability and model quality, while acknowledging limitations in generalization and privacy. Overall, the Label offers a scalable path toward standardized data documentation and governance to improve the outcomes of AI systems in practice.

Abstract

Artificial intelligence (AI) systems built on incomplete or biased data will often exhibit problematic outcomes. Current methods of data analysis, particularly before model development, are costly and not standardized. The Dataset Nutrition Label (the Label) is a diagnostic framework that lowers the barrier to standardized data analysis by providing a distilled yet comprehensive overview of dataset "ingredients" before AI model development. Building a Label that can be applied across domains and data types requires that the framework itself be flexible and adaptable; as such, the Label is comprised of diverse qualitative and quantitative modules generated through multiple statistical and probabilistic modelling backends, but displayed in a standardized format. To demonstrate and advance this concept, we generated and published an open source prototype with seven sample modules on the ProPublica Dollars for Docs dataset. The benefits of the Label are manyfold. For data specialists, the Label will drive more robust data analysis practices, provide an efficient way to select the best dataset for their purposes, and increase the overall quality of AI models as a result of more robust training datasets and the ability to check for issues at the time of model development. For those building and publishing datasets, the Label creates an expectation of explanation, which will drive better data collection practices. We also explore the limitations of the Label, including the challenges of generalizing across diverse datasets, and the risk of using "ground truth" data as a comparison dataset. We discuss ways to move forward given the limitations identified. Lastly, we lay out future directions for the Dataset Nutrition Label project, including research and public policy agendas to further advance consideration of the concept.

Figures (6)

• Figure 1: Model Development Pipeline
• Figure 2: (A) Survey results about data analysis best practices in respondents' organizations and (B) Survey results about how respondents learned to analyze data
• Figure 3: Architecture of the proposed Data Nutrition Label ecosystem.
• Figure 4: Prototype Label demonstrating the Pair Plot module and highlighting the interactive dropdown menus for selecting variables.
• Figure 5: Prototype Label demonstrating the Probabilistic Model module and showcasing a hypothetical distribution for payments made towards the drug "Eliquis" across different states.