WindGP: Efficient Graph Partitioning on Heterogenous Machines

TL;DR

WindGP addresses edge-partitioning for heterogeneous machines by formulating and minimizing a composite cost $TC$ that aggregates computation $T_i^{cal}$ and communication $T_i^{com}$ across machines. Its three-phase design—graph-oriented preprocessing to bound capacities, best-first expansion to build cohesive partitions within those bounds, and subgraph-local search to refine the result—yields partitions that significantly reduce distributed runtime compared to state-of-the-art methods. Across real-world and Graph 500 graphs, WindGP achieves $TC$ improvements in the range of roughly $1.35\times$ to $27\times$, and demonstrates scalability with graph size, machine count, and machine heterogeneity. These gains translate into practical performance improvements for large-scale distributed graph workloads such as PageRank and SSSP on skewed networks and mesh-like graphs, making WindGP a strong candidate for heterogeneous data-center environments.

Abstract

Graph Partitioning is widely used in many real-world applications such as fraud detection and social network analysis, in order to enable the distributed graph computing on large graphs. However, existing works fail to balance the computation cost and communication cost on machines with different power (including computing capability, network bandwidth and memory size), as they only consider replication factor and neglect the difference of machines in realistic data centers. In this paper, we propose a general graph partitioning algorithm WindGP, which can support fast and high-quality edge partitioning on heterogeneous machines. WindGP designs novel preprocessing techniques to simplify the metric and balance the computation cost according to the characteristics of graphs and machines. Also, best-first search is proposed instead of BFS and DFS, in order to generate clusters with high cohesion. Furthermore, WindGP adaptively tunes the partition results by sophisticated local search methods. Extensive experiments show that WindGP outperforms all state-of-the-art partition methods by 1.35 - 27 times on both dense and sparse distributed graph algorithms, and has good scalability with graph size and machine number.

Figures (15)

• Figure 1: The routine of BSP
• Figure 2: An example of graph partition
• Figure 3: The shortcoming of NE
• Figure 4: The framework of WindGP
• Figure 5: An example of vertex expansion

Theorems & Definitions (6)

• Definition 1
• Definition 2
• Definition 3
• Definition 4
• Lemma 1
• Theorem 1