KGLiDS: A Platform for Semantic Abstraction, Linking, and Automation of Data Science

TL;DR

KGLiDS presents a scalable, KG-based platform that interlinks data science datasets and pipelines through the LiDS ontology to enable data discovery and on-demand automation. It introduces a novel embedding-based data profiling (CoLR), static-dynamic pipeline abstraction, and Graph Neural Network (GNN)–driven data cleaning, transformation, and AutoML, all built on a RDF-star LiDS graph with a dedicated KG Governor. Empirical evaluations show substantial speedups and lower memory usage compared with state-of-the-art data discovery, data cleaning, and AutoML systems, while maintaining or improving accuracy. The work enables interoperable, reusable data science assets and lays groundwork for broader automation and knowledge sharing in data lakes and pipelines.

Abstract

In recent years, we have witnessed the growing interest from academia and industry in applying data science technologies to analyze large amounts of data. In this process, a myriad of artifacts (datasets, pipeline scripts, etc.) are created. However, there has been no systematic attempt to holistically collect and exploit all the knowledge and experiences that are implicitly contained in those artifacts. Instead, data scientists recover information and expertise from colleagues or learn via trial and error. Hence, this paper presents a scalable platform, KGLiDS, that employs machine learning and knowledge graph technologies to abstract and capture the semantics of data science artifacts and their connections. Based on this information, KGLiDS enables various downstream applications, such as data discovery and pipeline automation. Our comprehensive evaluation covers use cases in data discovery, data cleaning, transformation, and AutoML. It shows that KGLiDS is significantly faster with a lower memory footprint than the state-of-the-art systems while achieving comparable or better accuracy.

Figures (9)

• Figure 1: An overview of KGLiDS's main components: 1) KG Governor for pipeline abstraction, data profiling and KG construction as discussed in Section \ref{['sec:governor']}, 2) KGLiDS Storage for our RDF-star KG, embeddings and GNN models, 3) KGLiDS Interfaces, a Python library for different use cases, such as data discovery, cleaning, and transformation, as discussed in Section \ref{['sec:applications']}. KGLiDS supports predefined APIs and ad-hoc queries via SPARQL queries. Our library enables users to review recommended operations and execute them. KGLiDS enables automatic learning and discovery on open data science.
• Figure 2: An overview of the LiDS graph, which consists of the dataset, library, and pipeline graphs. Each pipeline is isolated in a named graph. Pipeline P1 is the abstraction of lines 2-10 in Figure \ref{['fig:running_example']}.
• Figure 3: A running example to demonstrate the KGLiDS Pipeline Abstraction. The pipeline loads a dataset using Pandas, performs data cleaning (imputation) and data transformation (scaling). The dataset is split into training and testing. Finally, a random forest classifier is fitted and evaluated.
• Figure 4: Top 10 libraries used in 13k Kaggle pipelines.
• Figure 5: Average precision and recall of unionable table discovery on all benchmark datasets.