CUTTANA: Scalable Graph Partitioning for Faster Distributed Graph Databases and Analytics

TL;DR

This work addresses partitioning massive graphs for distributed analytics and databases by marrying buffered streaming with a scalable refinement stage. CUTTANA buffering prevents premature placements and a coarsening/refinement process via a sub-partition graph yields higher-quality partitions while preserving latency comparable to streaming methods. Empirically, it delivers consistent improvements in edge-cut and communication volume across datasets, translating into up to 59% faster analytics and up to 23% higher graph-database throughput, with robust performance under edge-balance constraints. The approach highlights the importance of edge balance over vertex balance and offers a practical, parallelizable framework for large-scale graph partitioning.

Abstract

Graph partitioning plays a pivotal role in various distributed graph processing applications, including graph analytics, graph neural network training, and distributed graph databases. Graphs that require distributed settings are often too large to fit in the main memory of a single machine. This challenge renders traditional in-memory graph partitioners infeasible, leading to the emergence of streaming solutions. Streaming partitioners produce lower-quality partitions because they work from partial information and must make premature decisions before they have a complete view of a vertex's neighborhood. We introduce CUTTANA, a streaming graph partitioner that partitions massive graphs (Web/Twitter scale) with superior quality compared to existing streaming solutions. CUTTANA uses a novel buffering technique that prevents the premature assignment of vertices to partitions and a scalable coarsening and refinement technique that enables a complete graph view, improving the intermediate assignment made by a streaming partitioner. We implemented a parallel version for CUTTANA that offers nearly the same partitioning latency as existing streaming partitioners. Our experimental analysis shows that CUTTANA consistently yields better partitioning quality than existing state-of-the-art streaming vertex partitioners in terms of both edge-cut and communication volume metrics. We also evaluate the workload latencies that result from using CUTTANA and other partitioners in distributed graph analytics and databases. CUTTANA outperforms the other methods in most scenarios (algorithms, datasets). In analytics applications, CUTTANA improves runtime performance by up to 59% compared to various streaming partitioners (HDRF, Fennel, Ginger, HeiStream). In graph database tasks, CUTTANA results in higher query throughput by up to 23%, without hurting tail latency.

Figures (10)

• Figure 1: Partitioning a graph for two workers. Transferring data through edge-cuts (dotted edges) requires network calls.
• Figure 2: An example of partitioning's effect on network usage, compute imbalance, and total time of PageRank on the UK07 dataset. Cuttana improved PageRank execution time by more than 150s (52%) relative to Fenneltsourakakis2014fennel and (52s) 27% relative to HeiStreamfaraj2022buffered, while reducing partitioning time more than 50%.
• Figure 3: In large power-law graphs, the majority of vertices have low degrees ($\leq$ 1000) (left charts), and the majority of edges have at least one low-degree endpoint (right charts), even though the maximum degree in these networks exceeds a million.
• Figure 4: Partitioning of a graph and applying sub-partitioning and refinement with $\epsilon = 0.2$ balance condition and $\mathcal{K}'=5$.
• Figure 5: Parallel model of Cuttana execution.

Theorems & Definitions (12)

• Theorem 1
• proof
• Definition 1: Trade & Maximality
• Example 1
• Definition 2: Sub-Partitioning
• Definition 3: Sub-Partition Graph
• Proposition 1
• Example 2
• Lemma 1
• proof