Scalable community detection in massive networks via predictive assignment

TL;DR

It is proved that predictive assignment achieves strong consistency under the stochastic blockmodel and its degree-corrected version and the empirical performance of predictive assignment on simulated networks and two large real-world datasets are demonstrated.

Abstract

Massive network datasets are becoming increasingly common in scientific applications. Existing community detection methods encounter significant computational challenges for such massive networks due to two reasons. First, the full network needs to be stored and analyzed on a single server, leading to high memory costs. Second, existing methods typically use matrix factorization or iterative optimization using the full network, resulting in high runtimes. We propose a strategy called \textit{predictive assignment} to enable computationally efficient community detection while ensuring statistical accuracy. The core idea is to avoid large-scale matrix computations by breaking up the task into a smaller matrix computation plus a large number of vector computations that can be carried out in parallel. Under the proposed method, community detection is carried out on a small subgraph to estimate the relevant model parameters. Next, each remaining node is assigned to a community based on these estimates. We prove that predictive assignment achieves strong consistency under the stochastic blockmodel and its degree-corrected version. We also demonstrate the empirical performance of predictive assignment on simulated networks and two large real-world datasets: DBLP (Digital Bibliography \& Library Project), a computer science bibliographical database, and the Twitch Gamers Social Network.

Figures (3)

• Figure 1: A schema of the predictive assignment algorithm. Step 1: subsample selection; Step 2: community detection from the subgraph and estimation of the structural link parameter; Step 3: assignment of the remaining nodes to communities.
• Figure 2: Use of the different sections of the adjacency matrix under SBM (top panel) and DCBM (bottom panel). Here we have assumed, for the sake of simplicity, that $\mathcal{S} = \{1, \ldots, m\}$. For community detection in Step 2, $A_{(\mathcal{S},\mathcal{S})}$ (red border) is utilized under both models. Under the SBM, $A_{(\mathcal{S}^c,\mathcal{S})}$ (green border, top panel) is used to estimate $\Theta$. Under the DCBM, $A_{(\mathcal{S},\mathcal{S})}$ (red border, bottom panel) is used to estimate $\Omega$. Under both models, the blue-bordered vectors are used to assign the out-of-subgraph nodes to communities one by one in Step 3.
• Figure 3: Use of the different sections of the adjacency matrix using BASC under SBM for community detection in Step 2. Here we have assumed, for the sake of simplicity, that $\mathcal{S} = \{1, \ldots, m\}$. The submatrices $A_{(.,\mathcal{S})}$ (red border) and $A_{(\mathcal{S}^c,\mathcal{S})}$ (green border) are utilized for subgraph community detection and estimation of $\Theta$, respectively. The blue-bordered vectors represent $a_j$ and are used to assign nodes to communities one by one in Step 3.

Theorems & Definitions (11)

• Remark 2.1
• Remark 2.2
• Remark 2.3
• Remark 2.4
• Theorem 3.1
• Theorem 3.2
• Theorem 3.3
• Theorem 3.4
• Theorem 3.5
• Theorem 3.6