Performance of Distributed File Systems on Cloud Computing Environment: An Evaluation for Small-File Problem
Thanh Duong, Quoc Luu, Hung Nguyen
TL;DR
This paper investigates the performance of distributed file systems on small-file workloads within a unified architecture that combines high-performance computing with big data processing. It empirically compares Lustre and HDFS in Spark-based workflows using a small-file partitioned Criteo dataset across two clusters, demonstrating that Lustre can achieve higher throughput for large numbers of small files. The study highlights the small-file bottleneck as a key factor shaping design choices for shared storage in HPC and big data environments and suggests that Lustre is a viable unified storage backbone for complex machine learning tasks. Limitations include a small-scale evaluation and the absence of write benchmarks or Tachyon integration, guiding future work toward larger deployments and broader performance metrics.
Abstract
Various performance characteristics of distributed file systems have been well studied. However, the performance efficiency of distributed file systems on small-file problems with complex machine learning algorithms scenarios is not well addressed. In addition, demands for unified storage of big data processing and high-performance computing have been crucial. Hence, developing a solution combining high-performance computing and big data with shared storage is very important. This paper focuses on the performance efficiency of distributed file systems with small-file datasets. We propose an architecture combining both high-performance computing and big data with shared storage and perform a series of experiments to investigate the performance of these distributed file systems. The result of the experiments confirms the applicability of the proposed architecture in terms of complex machine learning algorithms.