Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Energy-Throughput Trade-off in Lossless-Compressed Source Code Storage

Paolo Ferragina, Francesco Tosoni

TL;DR

The paper tackles the challenge of efficiently indexing massive open-source code archives while minimizing energy use. It introduces a PPC-based compressed cache built on RocksDB as the last tier in a multi-tier storage system, enabling dynamic, scalable, lossless compression of source code data. Through micro-benchmarks and large-scale experiments on HPC hardware, it demonstrates Pareto-optimal trade-offs among space, time, and energy, showing that judicious compression and thread-parallelism can yield orders-of-magnitude improvements in retrieval throughput and energy efficiency. The work provides actionable guidelines for energy-aware configuration tuning and points to practical paths for greener storage backends in vast code repositories, with implications for AI training data pipelines and open-source code archiving.

Abstract

Retrieving data from large-scale source code archives is vital for AI training, neural-based software analysis, and information retrieval, to cite a few. This paper studies and experiments with the design of a compressed key-value store for the indexing of large-scale source code datasets, evaluating its trade-off among three primary computational resources: (compressed) space occupancy, time, and energy efficiency. Extensive experiments on a national high-performance computing infrastructure demonstrate that different compression configurations yield distinct trade-offs, with high compression ratios and order-of-magnitude gains in retrieval throughput and energy efficiency. We also study data parallelism and show that, while it significantly improves speed, scaling energy efficiency is more difficult, reflecting the known non-energy-proportionality of modern hardware and challenging the assumption of a direct time-energy correlation. This work streamlines automation in energy-aware configuration tuning and standardized green benchmarking deployable in CI/CD pipelines, thus empowering system architects with a spectrum of Pareto-optimal energy-compression-throughput trade-offs and actionable guidelines for building sustainable, efficient storage backends for massive open-source code archival.

The Energy-Throughput Trade-off in Lossless-Compressed Source Code Storage

TL;DR

The paper tackles the challenge of efficiently indexing massive open-source code archives while minimizing energy use. It introduces a PPC-based compressed cache built on RocksDB as the last tier in a multi-tier storage system, enabling dynamic, scalable, lossless compression of source code data. Through micro-benchmarks and large-scale experiments on HPC hardware, it demonstrates Pareto-optimal trade-offs among space, time, and energy, showing that judicious compression and thread-parallelism can yield orders-of-magnitude improvements in retrieval throughput and energy efficiency. The work provides actionable guidelines for energy-aware configuration tuning and points to practical paths for greener storage backends in vast code repositories, with implications for AI training data pipelines and open-source code archiving.

Abstract

Retrieving data from large-scale source code archives is vital for AI training, neural-based software analysis, and information retrieval, to cite a few. This paper studies and experiments with the design of a compressed key-value store for the indexing of large-scale source code datasets, evaluating its trade-off among three primary computational resources: (compressed) space occupancy, time, and energy efficiency. Extensive experiments on a national high-performance computing infrastructure demonstrate that different compression configurations yield distinct trade-offs, with high compression ratios and order-of-magnitude gains in retrieval throughput and energy efficiency. We also study data parallelism and show that, while it significantly improves speed, scaling energy efficiency is more difficult, reflecting the known non-energy-proportionality of modern hardware and challenging the assumption of a direct time-energy correlation. This work streamlines automation in energy-aware configuration tuning and standardized green benchmarking deployable in CI/CD pipelines, thus empowering system architects with a spectrum of Pareto-optimal energy-compression-throughput trade-offs and actionable guidelines for building sustainable, efficient storage backends for massive open-source code archival.
Paper Structure (15 sections, 5 figures, 2 tables)

This paper contains 15 sections, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Problem Definition and Results
  3. Basic Concepts
  4. Software Heritage
  5. Permute, Partition, and Compress
  6. Our Proposal
  7. Code Setup
  8. High-Performance Computing Infrastructure
  9. Datasets
  10. RocksDB Configuration
  11. Micro-Benchmarks on a 10 GiB Python Corpus
  12. Scaling up to the Large Collections
  13. Build phase
  14. Multithreaded retrieval
  15. Conclusions and Future Work

Figures (5)

  • Figure 1: A multi-tier storage architecture, which first directs key lookups to our PPC-based cache; it forwards queries to the Winery backend only upon a cache miss.
  • Figure 2: Software Heritage data model with source contents representing the leaves of the DAG
  • Figure 3: Our micro-benchmarks on 10-GiB Python code. The $y$-axis reports throughput (MiB/s for time, MB/J for energy); the $x$-axis shows the compression ratio. The best system configurations are in the top-left corner of each plot.
  • Figure 4: Time (dashed) and energy (solid) for querying the data in \ref{['tab:boffa_dataset']}.
  • Figure 5: Comparison amongst the best configurations for the four combined datasets in \ref{['tab:boffa_dataset']}.