Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A domain-specific language for describing machine learning datasets

Joan Giner-Miguelez, Abel Gómez, Jordi Cabot

TL;DR

The paper tackles the problem of opaque, informal, and hard-to-machine-process dataset documentation in ML, which can propagate biases and hinder replication. It proposes a domain-specific language with three core components—Metadata, Composition, and Provenance and Social Concerns—implemented as a Langium-based grammar inside a Visual Studio Code plugin to describe datasets in a machine-actionable way. A concrete metamodel and textual syntax are demonstrated on real datasets, with a preliminary evaluation showing that the DSL can capture existing documentation while also highlighting missing information in some cases. This work aims to enable automated analysis, comparison, and artifact generation for datasets, supporting a data-centric shift and potentially enabling dataset marketplaces and more trustworthy ML pipelines.

Abstract

Datasets play a central role in the training and evaluation of machine learning (ML) models. But they are also the root cause of many undesired model behaviors, such as biased predictions. To overcome this situation, the ML community is proposing a data-centric cultural shift where data issues are given the attention they deserve, and more standard practices around the gathering and processing of datasets start to be discussed and established. So far, these proposals are mostly high-level guidelines described in natural language and, as such, they are difficult to formalize and apply to particular datasets. In this sense, and inspired by these proposals, we define a new domain-specific language (DSL) to precisely describe machine learning datasets in terms of their structure, data provenance, and social concerns. We believe this DSL will facilitate any ML initiative to leverage and benefit from this data-centric shift in ML (e.g., selecting the most appropriate dataset for a new project or better replicating other ML results). The DSL is implemented as a Visual Studio Code plugin, and it has been published under an open source license.

A domain-specific language for describing machine learning datasets

TL;DR

The paper tackles the problem of opaque, informal, and hard-to-machine-process dataset documentation in ML, which can propagate biases and hinder replication. It proposes a domain-specific language with three core components—Metadata, Composition, and Provenance and Social Concerns—implemented as a Langium-based grammar inside a Visual Studio Code plugin to describe datasets in a machine-actionable way. A concrete metamodel and textual syntax are demonstrated on real datasets, with a preliminary evaluation showing that the DSL can capture existing documentation while also highlighting missing information in some cases. This work aims to enable automated analysis, comparison, and artifact generation for datasets, supporting a data-centric shift and potentially enabling dataset marketplaces and more trustworthy ML pipelines.

Abstract

Datasets play a central role in the training and evaluation of machine learning (ML) models. But they are also the root cause of many undesired model behaviors, such as biased predictions. To overcome this situation, the ML community is proposing a data-centric cultural shift where data issues are given the attention they deserve, and more standard practices around the gathering and processing of datasets start to be discussed and established. So far, these proposals are mostly high-level guidelines described in natural language and, as such, they are difficult to formalize and apply to particular datasets. In this sense, and inspired by these proposals, we define a new domain-specific language (DSL) to precisely describe machine learning datasets in terms of their structure, data provenance, and social concerns. We believe this DSL will facilitate any ML initiative to leverage and benefit from this data-centric shift in ML (e.g., selecting the most appropriate dataset for a new project or better replicating other ML results). The DSL is implemented as a Visual Studio Code plugin, and it has been published under an open source license.
Paper Structure (15 sections, 3 figures)

This paper contains 15 sections, 3 figures.

Table of Contents

  1. Introduction
  2. State of the art
  3. Data Documentation proposals from the ML community
  4. DSLs for datasets and ML
  5. DSL design
  6. Metadata
  7. Composition
  8. Provenance and Social Concerns
  9. DSL implementation and demonstration
  10. Metadata
  11. Composition
  12. Provenance and Social Concerns
  13. Preliminary evaluation
  14. Research Roadmap
  15. Conclusions

Figures (3)

  • Figure 1: Metadata model excerpt
  • Figure 2: Composition model excerpt
  • Figure 3: Provenance and Social Concerns model excerpt