Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Empowering Data Mesh with Federated Learning

Haoyuan Li, Salman Toor

TL;DR

This applied research article emphasizes the benefits of combining two distinct domains to achieve the best outcomes for industrial use cases and introduces a pioneering approach that incorporates Federated Learning into Data Mesh.

Abstract

The evolution of data architecture has seen the rise of data lakes, aiming to solve the bottlenecks of data management and promote intelligent decision-making. However, this centralized architecture is limited by the proliferation of data sources and the growing demand for timely analysis and processing. A new data paradigm, Data Mesh, is proposed to overcome these challenges. Data Mesh treats domains as a first-class concern by distributing the data ownership from the central team to each data domain, while keeping the federated governance to monitor domains and their data products. Many multi-million dollar organizations like Paypal, Netflix, and Zalando have already transformed their data analysis pipelines based on this new architecture. In this decentralized architecture where data is locally preserved by each domain team, traditional centralized machine learning is incapable of conducting effective analysis across multiple domains, especially for security-sensitive organizations. To this end, we introduce a pioneering approach that incorporates Federated Learning into Data Mesh. To the best of our knowledge, this is the first open-source applied work that represents a critical advancement toward the integration of federated learning methods into the Data Mesh paradigm, underscoring the promising prospects for privacy-preserving and decentralized data analysis strategies within Data Mesh architecture.

Empowering Data Mesh with Federated Learning

TL;DR

This applied research article emphasizes the benefits of combining two distinct domains to achieve the best outcomes for industrial use cases and introduces a pioneering approach that incorporates Federated Learning into Data Mesh.

Abstract

The evolution of data architecture has seen the rise of data lakes, aiming to solve the bottlenecks of data management and promote intelligent decision-making. However, this centralized architecture is limited by the proliferation of data sources and the growing demand for timely analysis and processing. A new data paradigm, Data Mesh, is proposed to overcome these challenges. Data Mesh treats domains as a first-class concern by distributing the data ownership from the central team to each data domain, while keeping the federated governance to monitor domains and their data products. Many multi-million dollar organizations like Paypal, Netflix, and Zalando have already transformed their data analysis pipelines based on this new architecture. In this decentralized architecture where data is locally preserved by each domain team, traditional centralized machine learning is incapable of conducting effective analysis across multiple domains, especially for security-sensitive organizations. To this end, we introduce a pioneering approach that incorporates Federated Learning into Data Mesh. To the best of our knowledge, this is the first open-source applied work that represents a critical advancement toward the integration of federated learning methods into the Data Mesh paradigm, underscoring the promising prospects for privacy-preserving and decentralized data analysis strategies within Data Mesh architecture.
Paper Structure (24 sections, 6 equations, 8 figures, 4 tables)

This paper contains 24 sections, 6 equations, 8 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Core Components
  4. Data Mesh
  5. Domain-Oriented Data Ownership
  6. Data as a Product
  7. Federated Computational Governance
  8. Federated Learning
  9. Horizontal Federated Learning
  10. Vertical Federated Learning
  11. Split Learning
  12. System Architecture
  13. Distributed Domain Data with Label Sharing
  14. Distributed Domain Data without Label Sharing
  15. Use Cases
  16. ...and 9 more sections

Figures (8)

  • Figure 1: The difference between horizontal federated learning(a), vertical federated learning(b) and split learning(c)
  • Figure 2: Basic Structure of Split Learning
  • Figure 3: Distributed Domain Data with Label Sharing
  • Figure 4: Distributed Domain Data without Label Sharing
  • Figure 5: Recommendation System for Retail Industry
  • ...and 3 more figures