Enterprise Data Science Platform: A Unified Architecture for Federated Data Access

TL;DR

The paper tackles data silos that arise when organizations need to share datasets across multiple analytics platforms. It proposes the Enterprise Data Science Platform (EDSP), a four-layer architecture built on a Write-Once, Read-Anywhere principle, enabling federated in-place access to a single data source via an Iceberg-based data store. Through production deployment and multi-engine evaluation (BigQuery, Snowflake, and Python), EDSP demonstrates a 33–44% reduction in data-access steps and practical query performance with at most a 2.6x latency overhead. The work provides concrete implementation guidelines, discusses challenges like metadata compatibility and access control, and offers a path toward vendor-neutral, scalable data sharing across diverse analytical environments.

Abstract

Organizations struggle to share data across departments that have adopted different data analytics platforms. If n datasets must serve m environments, up to n*m replicas can emerge, increasing inconsistency and cost. Traditional warehouses copy data into vendor-specific stores; cross-platform access is hard. This study proposes the Enterprise Data Science Platform (EDSP), which builds on data lakehouse architecture and follows a Write-Once, Read-Anywhere principle. EDSP enables federated data access for multi-query engine environments, targeting data science workloads with periodic data updates and query response times ranging from seconds to minutes. By providing centralized data management with federated access from multiple query engines to the same data sources, EDSP eliminates data duplication and vendor lock-in inherent in traditional data warehouses. The platform employs a four-layer architecture: Data Preparation, Data Store, Access Interface, and Query Engines. This design enforces separation of concerns and reduces the need for data migration when integrating additional analytical environments. Experimental results demonstrate that major cloud data warehouses and programming environments can directly query EDSP-managed datasets. We implemented and deployed EDSP in production, confirming interoperability across multiple query engines. For data sharing across different analytical environments, EDSP achieves a 33-44% reduction in operational steps compared with conventional approaches requiring data migration. Although query latency may increase by up to a factor of 2.6 compared with native tables, end-to-end completion times remain on the order of seconds, maintaining practical performance for analytical use cases. Based on our production experience, EDSP provides practical design guidelines for addressing the data-silo problem in multi-query engine environments.

Figures (2)

• Figure 1: From write-many silos to Write-Once, Read-Anywhere.(a) In a warehouse-centric setup, each source dataset $s_i$ is copied into each client/engine $c_j$, yielding up to $n\times m$ replicas $d_{ij}$ and siloed reads. (b) EDSP keeps one canonical table $t_i$ per source and exposes it to all engines via federated access, enabling Write-Once, Read-Anywhere; adding a new engine requires no data migration. Notation $S$, $D$, $T$, and $C$ follows the panel labels.
• Figure 2: Four-layer architecture of the Enterprise Data Science Platform (EDSP).