Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CuckooGraph: A Scalable and Space-Time Efficient Data Structure for Large-Scale Dynamic Graphs

Zhuochen Fan, Yalun Cai, Zirui Liu, Jiarui Guo, Xin Fan, Tong Yang, Bin Cui

TL;DR

CuckooGraph addresses the core challenges of large-scale dynamic graphs—rapid updates, massive scale, and complex queries—by adopting a transformable, hash-based storage that uses Large/Small Cuckoo Hash Tables (L-CHT/S-CHT) and a Denylist mechanism. The transformable data structures enable adaptive space reuse, while the Denylist mitigates insertion failures without frequent reconfigurations. The authors provide theoretical time and memory analyses and demonstrate substantial empirical gains over state-of-the-art baselines across insertion, query, and graph analytics tasks, with additional practicality through Redis and Neo4j integrations. The work offers a practical, scalable approach to dynamic graph storage with strong performance guarantees and broad applicability in real-world data systems.

Abstract

Graphs play an increasingly important role in various big data applications. However, existing graph data structures cannot simultaneously address the performance bottlenecks caused by the dynamic updates, large scale, and high query complexity of current graphs. This paper proposes a novel data structure for large-scale dynamic graphs called CuckooGraph. It does not require any prior knowledge of the upcoming graphs, and can adaptively resize to the most memory-efficient form while requiring few memory accesses for very fast graph data processing. The key techniques of CuckooGraph include TRANSFORMATION and DENYLIST. TRANSFORMATION fully utilizes the limited memory by designing related data structures that allow flexible space transformations to smoothly expand/tighten the required space depending on the number of incoming items. DENYLIST efficiently handles item insertion failures and further improves processing speed. Our experimental results show that compared with the most competitive solution Spruce, CuckooGraph achieves about $33\times$ higher insertion throughput while requiring only about $68\%$ of the memory space.

CuckooGraph: A Scalable and Space-Time Efficient Data Structure for Large-Scale Dynamic Graphs

TL;DR

CuckooGraph addresses the core challenges of large-scale dynamic graphs—rapid updates, massive scale, and complex queries—by adopting a transformable, hash-based storage that uses Large/Small Cuckoo Hash Tables (L-CHT/S-CHT) and a Denylist mechanism. The transformable data structures enable adaptive space reuse, while the Denylist mitigates insertion failures without frequent reconfigurations. The authors provide theoretical time and memory analyses and demonstrate substantial empirical gains over state-of-the-art baselines across insertion, query, and graph analytics tasks, with additional practicality through Redis and Neo4j integrations. The work offers a practical, scalable approach to dynamic graph storage with strong performance guarantees and broad applicability in real-world data systems.

Abstract

Graphs play an increasingly important role in various big data applications. However, existing graph data structures cannot simultaneously address the performance bottlenecks caused by the dynamic updates, large scale, and high query complexity of current graphs. This paper proposes a novel data structure for large-scale dynamic graphs called CuckooGraph. It does not require any prior knowledge of the upcoming graphs, and can adaptively resize to the most memory-efficient form while requiring few memory accesses for very fast graph data processing. The key techniques of CuckooGraph include TRANSFORMATION and DENYLIST. TRANSFORMATION fully utilizes the limited memory by designing related data structures that allow flexible space transformations to smoothly expand/tighten the required space depending on the number of incoming items. DENYLIST efficiently handles item insertion failures and further improves processing speed. Our experimental results show that compared with the most competitive solution Spruce, CuckooGraph achieves about higher insertion throughput while requiring only about of the memory space.
Paper Structure (33 sections, 4 theorems, 2 equations, 18 figures, 4 tables)

This paper contains 33 sections, 4 theorems, 2 equations, 18 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Our Proposed Solution
  4. Related Work
  5. Existing Solutions
  6. Orthogonal Work
  7. Cuckoo Hashing
  8. CuckooGraph Design
  9. Basic Version
  10. Foundation Stage (Transformable Data Structures)
  11. Optimization Stage (Denylist)
  12. Operations
  13. Extended Version
  14. Mathematical Analysis
  15. Time Cost of CuckooGraph
  16. ...and 18 more sections

Key Result

Theorem 1

Assume that a cuckoo hash table has $m$ buckets, each bucket has $d$ cells, and there are $n$ distinct items to insert. Let $dm=(1+\varepsilon)n$. If $d\geq \max \left\{ 8, 15.8\ln \frac{1}{\varepsilon} \right\}$, then the expected time complexity for inserting an item is $(\frac{1}{\varepsilon})^{O

Figures (18)

  • Figure 1: Data structure and examples of CuckooGraph. For clarity, only one hash table per L/S-CHT is shown.
  • Figure 2: Tuning experiments for parameter $d$.
  • Figure 3: Tuning experiments for parameter $G$.
  • Figure 4: Tuning experiments for parameter $T$.
  • Figure 5: Ablation experiments: CuckooGraph with and without DL optimization.
  • ...and 13 more figures

Theorems & Definitions (8)

  • Theorem 1
  • Theorem 2
  • proof
  • Definition 3
  • Lemma 4
  • proof
  • Theorem 5
  • proof