Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Weighted Similarity Metric for Community Detection in Sparse Data

Yong Zhang, Eric Herrison Gyamfi

TL;DR

The paper tackles the challenge of analyzing highly sparse short-text data by introducing a weighted similarity that avoids imputation. It combines three components—$S_{Num}$ for overlapping numerical values, $S_{Nan}$ for shared missingness, and $S_{Non}$ for non-matching missingness—into $\text{similarity} = S_{Num} + S_{Nan} + S_{Non}$, ensuring $S \in [-1,1]$. Through a shampoo-brand sentiment case study using a Girvan-Newman clustering setup, the method consistently outperforms imputation-based similarities across multiple structural and quality metrics, especially as network complexity grows. The approach is adaptable to different data types and domains, offering a robust framework for downstream analytics in sparse-data settings, with explicit emphasis on leveraging informative missingness without imputing values.

Abstract

Many Natural Language Processing (NLP) related applications involves topics and sentiments derived from short documents such as consumer reviews and social media posts. Topics and sentiments of short documents are highly sparse because a short document generally covers a few topics among hundreds of candidates. Imputation of missing data is sometimes hard to justify and also often unpractical in highly sparse data. We developed a method for calculating a weighted similarity for highly sparse data without imputation. This weighted similarity is consist of three components to capture similarities based on both existence and lack of common properties and pattern of missing values. As a case study, we used a community detection algorithm and this weighted similarity to group different shampoo brands based on sparse topic sentiments derived from short consumer reviews. Compared with traditional imputation and similarity measures, the weighted similarity shows better performance in both general community structures and average community qualities. The performance is consistent and robust across metrics and community complexities.

A Weighted Similarity Metric for Community Detection in Sparse Data

TL;DR

The paper tackles the challenge of analyzing highly sparse short-text data by introducing a weighted similarity that avoids imputation. It combines three components— for overlapping numerical values, for shared missingness, and for non-matching missingness—into , ensuring . Through a shampoo-brand sentiment case study using a Girvan-Newman clustering setup, the method consistently outperforms imputation-based similarities across multiple structural and quality metrics, especially as network complexity grows. The approach is adaptable to different data types and domains, offering a robust framework for downstream analytics in sparse-data settings, with explicit emphasis on leveraging informative missingness without imputing values.

Abstract

Many Natural Language Processing (NLP) related applications involves topics and sentiments derived from short documents such as consumer reviews and social media posts. Topics and sentiments of short documents are highly sparse because a short document generally covers a few topics among hundreds of candidates. Imputation of missing data is sometimes hard to justify and also often unpractical in highly sparse data. We developed a method for calculating a weighted similarity for highly sparse data without imputation. This weighted similarity is consist of three components to capture similarities based on both existence and lack of common properties and pattern of missing values. As a case study, we used a community detection algorithm and this weighted similarity to group different shampoo brands based on sparse topic sentiments derived from short consumer reviews. Compared with traditional imputation and similarity measures, the weighted similarity shows better performance in both general community structures and average community qualities. The performance is consistent and robust across metrics and community complexities.
Paper Structure (29 sections, 16 equations, 17 figures)

This paper contains 29 sections, 16 equations, 17 figures.

Table of Contents

  1. Introduction
  2. Imputation and Similarity Methods
  3. Imputation Methods
  4. Similarity Measures
  5. Formulation of the Weighted Similarity
  6. Similarity sNum
  7. Similarity sNan
  8. Similarity sNon
  9. Weighted Similarity
  10. Case Study
  11. Results
  12. Community Network Graph
  13. General Community Structures
  14. Average Community Qualities
  15. Discussion
  16. ...and 14 more sections

Figures (17)

  • Figure 1: How to calculate weighted similarity
  • Figure 2: Community structures with 5 clusters and 100 connected edges based on the weighted similarity and imputed cosine similarities using KNN.
  • Figure 3: Performance metrics on general community structures with 100 connected edges and 5 clusters. MQ: Modularity Q, C: Coverage, DI: Dunn Index, ACC: Average Clustering Coefficient, T: Transitivity, MD: Modularity Density, TPR: Triangle Participation Ratio. Higher bar indicates better community structure.
  • Figure 4: Average community qualities with 100 connected edges and 5 clusters. Higher bar indicates better community.
  • Figure 5: Average community qualities with 100 connected edges and 5 clusters. Shorter bar indicates better community.
  • ...and 12 more figures