Data Caching for Enterprise-Grade Petabyte-Scale OLAP

TL;DR

This paper tackles the challenge of compute-storage disaggregation in petabyte-scale OLAP by introducing Alluxio local cache, an embeddable edge cache that leverages local SSDs to reduce network I/O and improve data transfer efficiency between compute and storage. Through industrial traces and deployments at Uber and Meta, it analyzes workload characteristics such as data skew, fragmented small reads, and I/O bottlenecks, and presents an architecture featuring admission control, page-based storage, and hierarchical quotas. It provides two detailed case studies—Presto local cache and HDFS local cache—demonstrating significant production gains (e.g., reductions in P95 latency and I/O throttling) and offering actionable design lessons, such as 1 MB page sizes and robust metrics. The work demonstrates that a straightforward, well-integrated local cache can yield substantial performance, reliability, and cost benefits for large-scale OLAP ecosystems without disruptive platform-wide redesigns.

Abstract

With the exponential growth of data and evolving use cases, petabyte-scale OLAP data platforms are increasingly adopting a model that decouples compute from storage. This shift, evident in organizations like Uber and Meta, introduces operational challenges including massive, read-heavy I/O traffic with potential throttling, as well as skewed and fragmented data access patterns. Addressing these challenges, this paper introduces the Alluxio local (edge) cache, a highly effective architectural optimization tailored for such environments. This embeddable cache, optimized for petabyte-scale data analytics, leverages local SSD resources to alleviate network I/O and API call pressures, significantly improving data transfer efficiency. Integrated with OLAP systems like Presto and storage services like HDFS, the Alluxio local cache has demonstrated its effectiveness in handling large-scale, enterprise-grade workloads over three years of deployment at Uber and Meta. We share insights and operational experiences in implementing these optimizations, providing valuable perspectives on managing modern, massive-scale OLAP workloads.

Figures (16)

• Figure 1: Compute-storage disaggregation with SSD cache.
• Figure 2: Popularity rank and Zipfian distribution, indicating a heavy skewness in data access distribution.
• Figure 3: The workflow of Alluxio local cache.
• Figure 4: Organization of cached data files.
• Figure 5: Using indexed sets for organizing pages.