Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MapReplay: Trace-Driven Benchmark Generation for Java HashMap

Filippo Schiavio, Andrea Rosà, Júnior Löff, Lubomír Bulej, Petr Tůma, Walter Binder

Abstract

Hash-based maps, particularly java.util.HashMap, are pervasive in Java applications and the JVM, making their performance critical. Evaluating optimizations is challenging because performance depends on factors such as operation patterns, key distributions, and resizing behavior. Microbenchmarks are fast and repeatable but often oversimplify workloads, failing to capture the realistic usage patterns. Application benchmarks (e.g., DaCapo, Renaissance) provide realistic usages but are more expensive to run, prone to variability, and dominated by non-HashMap computations, making map-related performance changes difficult to observe. To address this challenge, we propose MapReplay, a benchmarking methodology that combines the realism of application benchmarks with the efficiency of microbenchmarks. MapReplay traces HashMap API usages generating a replay workload that reproduces the same operation sequence while faithfully reconstructing internal map states. This enables realistic and efficient evaluation of alternative implementations under realistic usage patterns. Applying MapReplay to DaCapo-Chopin and Renaissance, the resulting suite, MapReplayBench, reproduces application-level performance trends while reducing experimentation time and revealing insights difficult to obtain from full benchmarks.

MapReplay: Trace-Driven Benchmark Generation for Java HashMap

Abstract

Hash-based maps, particularly java.util.HashMap, are pervasive in Java applications and the JVM, making their performance critical. Evaluating optimizations is challenging because performance depends on factors such as operation patterns, key distributions, and resizing behavior. Microbenchmarks are fast and repeatable but often oversimplify workloads, failing to capture the realistic usage patterns. Application benchmarks (e.g., DaCapo, Renaissance) provide realistic usages but are more expensive to run, prone to variability, and dominated by non-HashMap computations, making map-related performance changes difficult to observe. To address this challenge, we propose MapReplay, a benchmarking methodology that combines the realism of application benchmarks with the efficiency of microbenchmarks. MapReplay traces HashMap API usages generating a replay workload that reproduces the same operation sequence while faithfully reconstructing internal map states. This enables realistic and efficient evaluation of alternative implementations under realistic usage patterns. Applying MapReplay to DaCapo-Chopin and Renaissance, the resulting suite, MapReplayBench, reproduces application-level performance trends while reducing experimentation time and revealing insights difficult to obtain from full benchmarks.
Paper Structure (66 sections, 3 figures, 3 tables)

This paper contains 66 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background: HashMap
  3. Design and Architecture
  4. Terminology and MapReplay Architecture
  5. Tracer.
  6. Offline trace post-processor.
  7. Replay infrastructure and replay workload.
  8. Goals
  9. Faithful execution paths.
  10. Equivalent map state.
  11. Design Decisions
  12. Operation categorization.
  13. Substitution of application keys, minimized replay of callbacks.
  14. Preserving hash collisions.
  15. Simplified equality semantics.
  16. ...and 51 more sections

Figures (3)

  • Figure 1: MapReplay architecture.
  • Figure 2: Microbenchmark performance for different operations, map sizes, and DIC configurations. Vertical gray lines indicate thresholds where resizing occurs.
  • Figure 3: Correlation between relative speedups (ratios) in application benchmarks and replay workloads. Different symbols indicate statistical significance of the combinations of speedups. Rectangles indicate the 99% bootstrap confidence intervals of the corresponding speedup ratios for weakly discordant changes plotted using the $\circ$ symbol.