Create Benchmarks for Data Lakes
Yi Lyu, Pei-Chieh Lo, Natan Lidukhover
TL;DR
This work addresses the lack of standardized benchmarks for data lakes that manage heterogeneous data. It proposes an extensible benchmarking framework that covers structured, semi-structured, and unstructured data, with workloads including retrieval, aggregation, querying, and similarity search, and metrics such as query execution time, metadata generation time, and metadata size. The framework leverages a graph-based keyword search (EASE) and uses datasets from IMDb, DBLP, and Apache logs to demonstrate reproducibility and scalability. Experimental results on local hardware reveal near-linear growth in graph-building time with data size and increasing query times, highlighting characteristic performance dynamics in data-lake workloads and guiding future benchmark development and platform comparisons.
Abstract
Data lakes have emerged as a flexible and scalable solution for storing and analyzing large volumes of heterogeneous data, including structured, semi-structured, and unstructured formats. Despite their growing adoption in both industry and academia, there is a lack of standardized and comprehensive benchmarks for evaluating the performance of data lake systems. Existing benchmarks primarily target traditional data warehouses and focus on structured SQL workloads, making them insufficient for capturing the diverse workloads and access patterns typical of data lakes. In this work, we propose a new benchmarking framework for data lakes that aims to provide an objective and comparative evaluation of different data lake implementations. Our benchmark covers multiple data types and workload models, including data retrieval, aggregation, querying, and similarity search, which is a common yet underexplored operation in existing benchmarks. We measure key performance metrics such as query execution time, metadata generation time, and metadata size across different scale factors. The benchmark is designed to be extensible and reproducible, enabling users to generate datasets and evaluate data lake systems under realistic and diverse scenarios. We conduct our experiments on CloudLab and demonstrate how the proposed benchmark can be used to compare both commercial and open-source data lake platforms.