Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Systematic Analysis on the Temporal Generalization of Language Models in Social Media

Asahi Ushio, Jose Camacho-Collados

TL;DR

This study analyzes how language models adapt to short-term temporal shifts in Twitter data by introducing a unified evaluation framework across five social media NLP tasks. It compares out-of-time and in-time training settings using multiple models and publicly available Twitter datasets, revealing consistent degradation for entity- and event-driven tasks (hate speech, NER, NED) under temporal shift, while topic and sentiment are less affected. Crucially, pre-training on data that includes the test period does not robustly mitigate the degradation, and capacity to adapt appears to hinge on understanding evolving entities rather than mere exposure to recent text. The work contributes a reproducible evaluation scheme, highlights the limits of naive temporal augmentation, and emphasizes the need for entity-aware context to improve robustness in fast-changing online content.

Abstract

In machine learning, temporal shifts occur when there are differences between training and test splits in terms of time. For streaming data such as news or social media, models are commonly trained on a fixed corpus from a certain period of time, and they can become obsolete due to the dynamism and evolving nature of online content. This paper focuses on temporal shifts in social media and, in particular, Twitter. We propose a unified evaluation scheme to assess the performance of language models (LMs) under temporal shift on standard social media tasks. LMs are tested on five diverse social media NLP tasks under different temporal settings, which revealed two important findings: (i) the decrease in performance under temporal shift is consistent across different models for entity-focused tasks such as named entity recognition or disambiguation, and hate speech detection, but not significant in the other tasks analysed (i.e., topic and sentiment classification); and (ii) continuous pre-training on the test period does not improve the temporal adaptability of LMs.

A Systematic Analysis on the Temporal Generalization of Language Models in Social Media

TL;DR

This study analyzes how language models adapt to short-term temporal shifts in Twitter data by introducing a unified evaluation framework across five social media NLP tasks. It compares out-of-time and in-time training settings using multiple models and publicly available Twitter datasets, revealing consistent degradation for entity- and event-driven tasks (hate speech, NER, NED) under temporal shift, while topic and sentiment are less affected. Crucially, pre-training on data that includes the test period does not robustly mitigate the degradation, and capacity to adapt appears to hinge on understanding evolving entities rather than mere exposure to recent text. The work contributes a reproducible evaluation scheme, highlights the limits of naive temporal augmentation, and emphasizes the need for entity-aware context to improve robustness in fast-changing online content.

Abstract

In machine learning, temporal shifts occur when there are differences between training and test splits in terms of time. For streaming data such as news or social media, models are commonly trained on a fixed corpus from a certain period of time, and they can become obsolete due to the dynamism and evolving nature of online content. This paper focuses on temporal shifts in social media and, in particular, Twitter. We propose a unified evaluation scheme to assess the performance of language models (LMs) under temporal shift on standard social media tasks. LMs are tested on five diverse social media NLP tasks under different temporal settings, which revealed two important findings: (i) the decrease in performance under temporal shift is consistent across different models for entity-focused tasks such as named entity recognition or disambiguation, and hate speech detection, but not significant in the other tasks analysed (i.e., topic and sentiment classification); and (ii) continuous pre-training on the test period does not improve the temporal adaptability of LMs.
Paper Structure (23 sections, 10 figures, 6 tables)

This paper contains 23 sections, 10 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LMs on Social Media.
  4. Temporal Generalization.
  5. Temporal-aware LMs.
  6. Experimental Setting
  7. Evaluation Methodology
  8. Out-of-Time (OOT).
  9. In-Time (IT).
  10. Tasks & Datasets
  11. Hate Speech Detection.
  12. Topic Classification.
  13. Sentiment Analysis.
  14. Named Entity Disambiguation (NED).
  15. Named Entity Recognition (NER).
  16. ...and 8 more sections

Figures (10)

  • Figure 1: An illustrative example of the conceptual differences between the sampling time periods of the OOT and IT settings.
  • Figure 2: Quarterly breakdown of the number of tweets ratio (%) in each dataset. For example, a ratio of 5% in 13-Q3 for Dataset X would mean that 5% of all tweets in Dataset X belong to the third quarter (July-September) of 2013.
  • Figure 3: Comparisons of ratio (%) of positive labels in the training split of each task between OOT and IT.
  • Figure 4: Comparisons of label distributions between OOT and IT settings.
  • Figure 5: Comparisons of IT and OOT performance (accuracy) for hate speech detection.
  • ...and 5 more figures