Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Online Density-Based Clustering for Real-Time Narrative Evolution Monitorin

Ostap Vykhopen, Viktoria Skorik, Maxim Tereschenko, Veronika Solopova

TL;DR

The paper tackles the bottlenecks of batch clustering in real-time narrative monitoring and questions whether online density-based clustering can replace HDBSCAN in production pipelines. It introduces a streaming evaluation framework and compares online algorithms (DenStream, DBSTREAM) against HDBSCAN within a multilingual, embedding-based topic discovery pipeline, highlighting both cluster-quality improvements and temporal stability trade-offs. Key findings show DenStream offers superior standard clustering metrics and competitive narrative distinctness, while HDBSCAN remains more temporally stable; DBSTREAM performs poorly on high-dimensional semantic space. The work demonstrates that online density-based clustering can be viable for crisis informatics and narrative surveillance, provided system design accounts for stability vs. quality and downstream labeling alignment, with future work focusing on stabilization and retuning downstream components.

Abstract

Automated narrative intelligence systems for social media monitoring face significant scalability challenges when processing continuous data streams using traditional batch clustering algorithms. We investigate the replacement of HDBSCAN (offline clustering) with online (streaming/incremental) clustering methods in a production narrative report generation pipeline. The proposed system employs a three-stage architecture (data collection, modeling, dashboard generation) that processes thousands of multilingual social media documents daily. While HDBSCAN excels at discovering hierarchical density-based clusters and handling noise, its batch-only nature necessitates complete retraining for each time window, resulting in memory constraints, computational inefficiency, and inability to adapt to evolving narratives in real-time. This work evaluates a bunch of online clustering algorithms across dimensions of cluster quality preservation, computational efficiency, memory footprint, and integration compatibility with existing workflows. We propose evaluation criteria that balance traditional clustering metrics (Silhouette Coefficient, Davies-Bouldin Index) with narrative metrics (narrative distinctness, contingency and variance). Our methodology includes sliding-window simulations on historical datasets from Ukraine information space, enabling comparative analysis of algorithmic trade-offs in realistic operational contexts. This research addresses a critical gap between batch-oriented topic modeling frameworks and the streaming nature of social media monitoring, with implications for computational social science, crisis informatics, and narrative surveillance systems.

Online Density-Based Clustering for Real-Time Narrative Evolution Monitorin

TL;DR

The paper tackles the bottlenecks of batch clustering in real-time narrative monitoring and questions whether online density-based clustering can replace HDBSCAN in production pipelines. It introduces a streaming evaluation framework and compares online algorithms (DenStream, DBSTREAM) against HDBSCAN within a multilingual, embedding-based topic discovery pipeline, highlighting both cluster-quality improvements and temporal stability trade-offs. Key findings show DenStream offers superior standard clustering metrics and competitive narrative distinctness, while HDBSCAN remains more temporally stable; DBSTREAM performs poorly on high-dimensional semantic space. The work demonstrates that online density-based clustering can be viable for crisis informatics and narrative surveillance, provided system design accounts for stability vs. quality and downstream labeling alignment, with future work focusing on stabilization and retuning downstream components.

Abstract

Automated narrative intelligence systems for social media monitoring face significant scalability challenges when processing continuous data streams using traditional batch clustering algorithms. We investigate the replacement of HDBSCAN (offline clustering) with online (streaming/incremental) clustering methods in a production narrative report generation pipeline. The proposed system employs a three-stage architecture (data collection, modeling, dashboard generation) that processes thousands of multilingual social media documents daily. While HDBSCAN excels at discovering hierarchical density-based clusters and handling noise, its batch-only nature necessitates complete retraining for each time window, resulting in memory constraints, computational inefficiency, and inability to adapt to evolving narratives in real-time. This work evaluates a bunch of online clustering algorithms across dimensions of cluster quality preservation, computational efficiency, memory footprint, and integration compatibility with existing workflows. We propose evaluation criteria that balance traditional clustering metrics (Silhouette Coefficient, Davies-Bouldin Index) with narrative metrics (narrative distinctness, contingency and variance). Our methodology includes sliding-window simulations on historical datasets from Ukraine information space, enabling comparative analysis of algorithmic trade-offs in realistic operational contexts. This research addresses a critical gap between batch-oriented topic modeling frameworks and the streaming nature of social media monitoring, with implications for computational social science, crisis informatics, and narrative surveillance systems.
Paper Structure (19 sections, 2 figures, 2 tables)

This paper contains 19 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Theoretical Background and Related work
  3. Density-Based Clustering
  4. Online and Streaming Clustering Algorithms
  5. Streaming Topic Modeling
  6. Methodology & Experimental setup
  7. System Architecture
  8. Data Collection and Preprocessing.
  9. Modeling Pipeline.
  10. Experiment Design
  11. Dataset
  12. Baseline System.
  13. Online Algorithm Evaluation.
  14. Evaluation Metrics.
  15. Results
  16. ...and 4 more sections

Figures (2)

  • Figure 1: Four stage pipeline for topic and narrative discovery.
  • Figure 2: Comparison of online clustering methods: (a) HDBSCAN, (b) DBSTREAM, and (c) DenStream. Each fingerprint represents a cluster projected onto a set of domain-specific keywords, forming a polygon that encodes its semantic profile. Sharp spikes indicate focused narratives, while broad shapes show multi-topic or hybrid narratives. Overlapping areas reveal shared or competing narratives, and may also indicate overlapping clustering. The charts above show the Semantic Fingerprints of three clusters, based on a set of highly relevant keywords in the current Ukrainian information space. Chart (A) (HDBSCAN) shows that the first cluster covers nearly all keywords, which may indicate contamination by a mix of different narratives. In contrast, DBSTREAM and DenStream display a clearer per-keyword distribution and better separation between clusters.