Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Metadata-driven Table Union Search: Leveraging Semantics for Restricted Access Data Integration

Margherita Martorana, Tobias Kuhn, Jacco van Ossenbruggen

TL;DR

This paper addresses the challenge of performing Table Union Search on restricted access data by relying exclusively on metadata signals. It introduces Metadata Union Search (MUS), which enriches column-level metadata with semantic data types and DBpedia properties, and computes unionability through embeddings and cosine similarity, considering both topic-dependent and topic-guided settings. On the ALT-gen benchmark, MUS achieves strong performance and outperforms several baselines in precision and recall, with MUSt thresholds around $0.63$-$0.69$ for TD and $0.79$-$0.84$ for TG, and an overall claim of up to 81% accuracy in unionability. The work demonstrates that metadata-driven approaches can enable Findable, Accessible, Interoperable, and Reusable data discovery for privacy-sensitive domains, offering practical impact for secure data integration without exposing confidential values.

Abstract

Over the past decade, the Table Union Search (TUS) task has aimed to identify unionable tables within data lakes to improve data integration and discovery. While numerous solutions and approaches have been introduced, they primarily rely on open data, making them not applicable to restricted access data, such as medical records or government statistics, due to privacy concerns. Restricted data can still be shared through metadata, which ensures confidentiality while supporting data reuse. This paper explores how TUS can be computed on restricted access data using metadata alone. We propose a method that achieves 81% accuracy in unionability and outperforms existing benchmarks in precision and recall. Our results highlight the potential of metadata-driven approaches for integrating restricted data, facilitating secure data discovery in privacy-sensitive domains. This aligns with the FAIR principles, by ensuring data is Findable, Accessible, Interoperable, and Reusable while preserving confidentiality.

Metadata-driven Table Union Search: Leveraging Semantics for Restricted Access Data Integration

TL;DR

This paper addresses the challenge of performing Table Union Search on restricted access data by relying exclusively on metadata signals. It introduces Metadata Union Search (MUS), which enriches column-level metadata with semantic data types and DBpedia properties, and computes unionability through embeddings and cosine similarity, considering both topic-dependent and topic-guided settings. On the ALT-gen benchmark, MUS achieves strong performance and outperforms several baselines in precision and recall, with MUSt thresholds around - for TD and - for TG, and an overall claim of up to 81% accuracy in unionability. The work demonstrates that metadata-driven approaches can enable Findable, Accessible, Interoperable, and Reusable data discovery for privacy-sensitive domains, offering practical impact for secure data integration without exposing confidential values.

Abstract

Over the past decade, the Table Union Search (TUS) task has aimed to identify unionable tables within data lakes to improve data integration and discovery. While numerous solutions and approaches have been introduced, they primarily rely on open data, making them not applicable to restricted access data, such as medical records or government statistics, due to privacy concerns. Restricted data can still be shared through metadata, which ensures confidentiality while supporting data reuse. This paper explores how TUS can be computed on restricted access data using metadata alone. We propose a method that achieves 81% accuracy in unionability and outperforms existing benchmarks in precision and recall. Our results highlight the potential of metadata-driven approaches for integrating restricted data, facilitating secure data discovery in privacy-sensitive domains. This aligns with the FAIR principles, by ensuring data is Findable, Accessible, Interoperable, and Reusable while preserving confidentiality.

Paper Structure

This paper contains 17 sections, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Table Union Search
  4. Data Integration through Metadata-Driven Approaches
  5. Metadata Union Search
  6. Experimental Design
  7. Data Preparation
  8. Metadata Enrichment
  9. Computing Unionability on Metadata
  10. Evaluation
  11. Results
  12. Metadata Creation and Enrichment
  13. Metadata Union Search
  14. Comparison of Unionability Methods
  15. Discussion
  16. ...and 2 more sections

Figures (1)

  • Figure 1: Excerpt of a CSV dataset from the ALT-gen benchmark, with its corresponding metadata. In the metadata we can see the enrichments, added with the dcterms:type property for the semantic data types, and with the dsv:columnProperty property for the DBpedia properties.