Memory-Efficient Community Detection on Large Graphs Using Weighted Sketches

TL;DR

This work tackles the memory bottleneck of parallel community detection on large graphs by replacing per-thread collision-free hashtables with weighted Misra-Gries sketches in Louvain, Leiden, and LPA. The MG-based approach dramatically reduces memory usage while incurring only a small drop in modularity and moderate runtime overhead, enhancing parallel scalability on shared-memory systems. Across 13 large real-world graphs, the MG-enabled methods achieve competitive quality with substantial memory savings, confirming the practicality of memory-efficient sketches for large-scale graph clustering. The proposed technique has significant implications for deploying community detection on systems with many cores, offering a pathway to surpass memory-intensive methods as thread counts grow.

Abstract

Community detection in graphs identifies groups of nodes with denser connections within the groups than between them, and while existing studies often focus on optimizing detection performance, memory constraints become critical when processing large graphs on shared-memory systems. We recently proposed efficient implementations of the Louvain, Leiden, and Label Propagation Algorithms (LPA) for community detection. However, these incur significant memory overhead from the use of collision-free per-thread hashtables. To address this, we introduce memory-efficient alternatives using weighted Misra-Gries (MG) sketches, which replace the per-thread hashtables, and reduce memory demands in Louvain, Leiden, and LPA implementations - while incurring only a minor quality drop (up to 1%) and moderate runtime penalties. We believe that these approaches, though slightly slower, are well-suited for parallel processing and could outperform current memory-intensive techniques on systems with many threads.

Figures (12)

• Figure 1: Illustration of collision-free, per-thread hashtables (Far-KV) that are well spaced in memory, for two threads. Each hashtable includes a keys list, a values array (of size $|V|$), and a key count ($N_0$ and $N_1$). Values associated with each key are stored and accumulated at the index specified by the key. To prevent false cache sharing, the key counts for each hashtable are independently allocated on the heap.
• Figure 2: Illustration of our modification to the local-moving phase of the Louvain algorithm, the local-moving and refinement phases of the Leiden algorithm, and the iterations of LPA. Here, $N_1$, $N_2$, $N_3$, $N_4$ denote the neighbors of a vertex $i$ in the graph, with associated edge weights to $i$ of $w_1$, $w_2$, $w_3$, $w_4$, respectively, while $C_a$, $C_b$, $C_c$ denote the community memberships of the neighbors. This is used to populate a weighted Misra-Gries (MG) sketch with $k$ slots, where $V_a$, $V_b$, and $V_c$ in the MG sketch represent the accumulated weights for communities $C_a$, $C_b$, and $C_c$, respectively. Once populated, a second pass is made over vertex $i$’s edges to compute the total linking weights $W_a$, $W_b$, and $W_c$ for the majority candidate communities $C_a$, $C_b$, and $C_c$ --- which is then used to compute the delta-modularities $\Delta Q_a$, $\Delta Q_b$, $\Delta Q_c$ of moving $i$ to the candidate communities $C_a$, $C_b$, $C_c$, respectively. Finally, the community $C_b$ yielding the highest positive modularity gain $\Delta Q_b$ is selected as $i$’s new community.
• Figure 3: Illustration of our modification of the aggregation phase in the Louvain and Leiden algorithms. Here, $N_1$ and $N_2$ are neighbors of a vertex in community $c$, while $N_6$ and $N_7$ are neighbors of another vertex in the same community --- with associated edge weights $w_1$, $w_2$, $w_6$, and $w_7$ to their respective source vertex, and with associated community memberships $C_a$, $C_b$, $C_c$, and $C_d$. These are used to populate a weighted Misra-Gries (MG) sketch with $k$ slots, where $V_a$, $V_b$, $V_c$, and $V_d$ in the MG sketch represent the cumulative weights for communities $C_a$, $C_b$, $C_c$, and $C_d$, respectively. Once populated, a super-vertex $c$ is created in the aggregated graph, linking to each community $C_a$, $C_b$, $C_c$, and $C_d$ with the corresponding weights $V_a$, $V_b$, $V_c$, and $V_d$. To prevent disconnected communities, our modified Leiden also ensures that the aggregated graph remains undirected.
• Figure 4: Relative Runtime and Relative Modularity of communities obtained from Default, Boyer-Moore (BM), and Misra-Gries (MG) based Louvain, with slot counts ranging from $4$ to $256$ in powers of $2$. Two variations of each MG-based Louvain are compared: one that unconditionally subtracts values from all non-matching keys before inserting a new key-value pair into the Misra-Gries sketch, and another that performs conditional subtraction only after a failed insertion attempt. Although Default Louvain has only one method, it is shown as two variations for simplicity. The most suitable approach is highlighted.
• Figure 5: Relative Runtime and Relative Modularity of communities from Default, Boyer-Moore (BM), and Misra-Gries (MG) based Leiden, with the number of slots ranging from $4$ to $256$ in powers of $2$. Two variations of each MG-based Leiden are compared: one that unconditionally subtracts values from all non-matching keys before inserting a new key-value pair into the MG sketch, and another that performs conditional subtraction only after a failed insertion attempt. Although Default Leiden is a single method, it is shown as two variations for simplicity. The most suitable approach is highlighted in the figure.