Dynamic hashtag recommendation in social media with trend shift detection and adaptation

TL;DR

The paper tackles the problem that static hashtag recommendations fail to keep up with rapidly shifting social media trends. It introduces H-ADAPTS, a trend-aware dynamic recommender built on HASHET, augmented with real-time trend shift detection via the Ranked Jaccard Distance and an asynchronous, windowed adaptation pipeline; DistilBERT reduces training time to support rapid re-alignment, while Apache Storm handles unbounded data streams. Key contributions include the trend shift detector, an efficient embedding-space and mapper adaptation process over sliding windows, and a comparative evaluation on COVID-19 and the 2020 US elections showing robust, timely hashtag recommendations that outperform static and non-adaptive baselines. The approach has practical impact for real-time social data analytics, enabling more relevant labeling and retrieval in fast-moving conversations.

Abstract

Hashtag recommendation systems have emerged as a key tool for automatically suggesting relevant hashtags and enhancing content categorization and search. However, existing static models struggle to adapt to the highly dynamic nature of social media conversations, where new hashtags constantly emerge and existing ones undergo semantic shifts. To address these challenges, this paper introduces H-ADAPTS (Hashtag recommendAtion by Detecting and adAPting to Trend Shifts), a dynamic hashtag recommendation methodology that employs a trend-aware mechanism to detect shifts in hashtag usage-reflecting evolving trends and topics within social media conversations-and triggers efficient model adaptation based on a (small) set of recent posts. Additionally, the Apache Storm framework is leveraged to support scalable and fault-tolerant analysis of high-velocity social data, enabling the timely detection of trend shifts. Experimental results from two real-world case studies, including the COVID-19 pandemic and the 2020 US presidential election, demonstrate the effectiveness of H-ADAPTS in providing timely and relevant hashtag recommendations by adapting to emerging trends, significantly outperforming existing solutions.

Figures (7)

• Figure 1: Execution flow of H-ADAPTS comprising four steps: $(1)$model bootstrap, $(2)$trend shift detection, $(3)$model adaptation, and $(4)$hashtag recommendation.
• Figure 2: Storm topology supporting the execution of H-ADAPTS given an unbounded stream of social media posts.
• Figure 3: Comparison of weekly average recall among the different alternative strategies and the proposed one, i.e., TL($MLP$, $W$) + FT($E$$+$$MLP$, $F$).
• Figure 4: Point-wise daily comparison between H-ADAPTS and HASHET for the COVID-19 pandemic case study. Trend shifts are indicated by vertical dotted lines.
• Figure 5: Comparison with related techniques over time for the COVID-19 pandemic case study, in terms of average recall. Vertical dotted lines indicate trend shifts and corresponding adaptations by H-ADAPTS.