Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RadixGraph: A Fast, Space-Optimized Data Structure for Dynamic Graph Storage (Extended Version)

Haoxuan Xie, Junfeng Liu, Siqiang Luo, Kai Wang

TL;DR

RadixGraph tackles the challenge of in‑memory dynamic graph storage by introducing a space‑optimal vertex index (SORT) and a snapshot–log edge storage that jointly deliver fast updates and compact memory usage. The SORT index uses a data‑driven optimization to pick per‑layer fan‑outs in an l‑layer radix tree, achieving near‑linear space with $O(l)$ lookup costs, while the edge storage splits adjacency into a read‑only snapshot and an append‑only log, enabling amortized $O(1)$ edge updates via log compaction and producing an edge chain that supports traversal with minimal vertex lookups. The design is backed by MVCC support, latch‑free log appends, and a dynamic programming framework for SORT configuration, with empirical results showing up to $16.27\times$ faster updates and a $40.1\%$ memory reduction relative to strong baselines across real and synthetic graphs. Together, these innovations offer scalable, concurrent graph storage and analytics for very large dynamic graphs, with open‑source availability at the RadixGraph repository.

Abstract

Dynamic graphs model many real-world applications, and as their sizes grow, efficiently storing and updating them becomes critical. We present RadixGraph, a fast and memory-efficient data structure for dynamic graph storage. RadixGraph features a carefully designed radix-tree-based vertex index that strikes an optimal trade-off between query efficiency and space among all pointer-array-based radix trees. For edge storage, it employs a hybrid snapshot-log architecture that enables amortized $O(1)$ update time. RadixGraph supports millions of concurrent updates per second while maintaining competitive performance for graph analytics. Experimental results show that RadixGraph outperforms the most performant baseline by up to $16.27\times$ across various datasets in ingesting graph updates, and reduces memory usage by an average of $40.1\%$. RadixGraph is open-source at https://github.com/ForwardStar/RadixGraph.

RadixGraph: A Fast, Space-Optimized Data Structure for Dynamic Graph Storage (Extended Version)

TL;DR

RadixGraph tackles the challenge of in‑memory dynamic graph storage by introducing a space‑optimal vertex index (SORT) and a snapshot–log edge storage that jointly deliver fast updates and compact memory usage. The SORT index uses a data‑driven optimization to pick per‑layer fan‑outs in an l‑layer radix tree, achieving near‑linear space with lookup costs, while the edge storage splits adjacency into a read‑only snapshot and an append‑only log, enabling amortized edge updates via log compaction and producing an edge chain that supports traversal with minimal vertex lookups. The design is backed by MVCC support, latch‑free log appends, and a dynamic programming framework for SORT configuration, with empirical results showing up to faster updates and a memory reduction relative to strong baselines across real and synthetic graphs. Together, these innovations offer scalable, concurrent graph storage and analytics for very large dynamic graphs, with open‑source availability at the RadixGraph repository.

Abstract

Dynamic graphs model many real-world applications, and as their sizes grow, efficiently storing and updating them becomes critical. We present RadixGraph, a fast and memory-efficient data structure for dynamic graph storage. RadixGraph features a carefully designed radix-tree-based vertex index that strikes an optimal trade-off between query efficiency and space among all pointer-array-based radix trees. For edge storage, it employs a hybrid snapshot-log architecture that enables amortized update time. RadixGraph supports millions of concurrent updates per second while maintaining competitive performance for graph analytics. Experimental results show that RadixGraph outperforms the most performant baseline by up to across various datasets in ingesting graph updates, and reduces memory usage by an average of . RadixGraph is open-source at https://github.com/ForwardStar/RadixGraph.
Paper Structure (26 sections, 4 theorems, 17 equations, 14 figures, 7 tables, 2 algorithms)

This paper contains 26 sections, 4 theorems, 17 equations, 14 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Radix tree and its variants
  4. Existing solutions and limitations
  5. Design of RadixGraph
  6. SORT: a Space-Optimized Radix Tree
  7. Optimizing SORT configuration
  8. Snapshot-log edge storage
  9. Discussions
  10. Experiments
  11. Setup
  12. Dynamic operations
  13. Memory consumption
  14. Graph analytics
  15. Concurrent reads and writes
  16. ...and 11 more sections

Key Result

Lemma 1

(Proof in Appendix A) There exists an optimal solution $(s_0^*,s_1^*,\cdots,s_{l-1}^*)$, such that $s_{l-1}^*=x$.

Figures (14)

  • Figure 1: Different radix tree structures for storing 4-bit integers. Only integers 3 (i.e., 0011) and 6 (i.e., 0110) are inserted.
  • Figure 2: The high-level overview of RadixGraph. Parenthesis "(x)" represents an offset corresponding to the location in the vertex table. The fan-out size $2^{a_i}$ is the pointer array size of a node at layer $i$ and is pre-determined by our optimizer. In this example, $a_0=3,a_1=2,a_2=1$.
  • Figure 3: The data layout of vertex, edge and log blocks.
  • Figure 4: The representation intervals of $i$-layer nodes.
  • Figure 5: Example edge update and compaction processes. Offset (x) corresponds to the bit $B[(x/32)/L)][(x/32)\text{ mod } L]$.
  • ...and 9 more figures

Theorems & Definitions (4)

  • Lemma 1
  • Theorem 1
  • Theorem 2
  • Theorem 3