BYO: A Unified Framework for Benchmarking Large-Scale Graph Containers

TL;DR

BYO presents a unified, apples-to-apples framework for benchmarking large-scale graph containers, decoupling container performance from algorithm and infrastructure factors. By integrating 27 containers across 10 algorithms and 10 graphs within a GBBS-based translation layer, it reveals that average container performance differences are modest, with off-the-shelf structures approaching CSR performance and minimal API configurations nearly matching full APIs. The study shows that inline optimization and data locality improvements yield measurable gains, while compression offers limited benefits at large scales. Overall, BYO formalizes container benchmarking, enabling fair comparisons, guiding container design, and highlighting scenarios where specialized data structures provide the most benefit, especially on challenging graphs and updates.

Abstract

A fundamental building block in any graph algorithm is a graph container - a data structure used to represent the graph. Ideally, a graph container enables efficient access to the underlying graph, has low space usage, and supports updating the graph efficiently. In this paper, we conduct an extensive empirical evaluation of graph containers designed to support running algorithms on large graphs. To our knowledge, this is the first apples-to-apples comparison of graph containers rather than overall systems, which include confounding factors such as differences in algorithm implementations and infrastructure. We measure the running time of 10 highly-optimized algorithms across over 20 different containers and 10 graphs. Somewhat surprisingly, we find that the average algorithm running time does not differ much across containers, especially those that support dynamic updates. Specifically, a simple container based on an off-the-shelf B-tree is only 1.22x slower on average than a highly optimized static one. Moreover, we observe that simplifying a graph-container Application Programming Interface (API) to only a few simple functions incurs a mere 1.16x slowdown compared to a complete API. Finally, we also measure batch-insert throughput in dynamic-graph containers for a full picture of their performance. To perform the benchmarks, we introduce BYO, a unified framework that standardizes evaluations of graph-algorithm performance across different graph containers. BYO extends the Graph Based Benchmark Suite (Dhulipala et al. 18), a state-of-the-art graph algorithm benchmark, to easily plug into different dynamic graph containers and enable fair comparisons between them on a large suite of graph algorithms. While several graph algorithm benchmarks have been developed to date, to the best of our knowledge, BYO is the first system designed to benchmark graph containers

Figures (6)

• Figure 1: Relationship between BYO, graph-algorithm frameworks, and dynamic-graph containers.
• Figure 2: Slowdown of each container compared to CSR. Each point ($x, y$) for a given graph containermeans that the container was at most $x$ times slower than CSR on $y\%$ of experiments. A line going up faster implies that the container achieves closer performance relative to CSR on more experiments. We find that almost all structures are able to perform the majority of the experiments with at most a $1.4\times$ slowdown over CSR. std::set and absl::* are off-the-shelf containers, while the others are optimized. Details on the graph containers can be found in Section \ref{['sec:datastr']}.
• Figure 3: The throughput of inserts for different batch sizes. Of the data structures in this plot, std::set and absl::*, are off-the-shelf containers, while the others are optimized. Details on the different graph containers can be found in Section \ref{['sec:datastr']}.
• Figure 4: Relationship between BYO framework, graph containers, and graph algorithms (via GBBS).
• Figure 5: Data structure and inline optimization using the set API.