Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Methods for Generating Drift in Text Streams

Cristiano Mesquita Garcia, Alessandro Lameiras Koerich, Alceu de Souza Britto, Jean Paul Barddal

TL;DR

Four textual drift generation methods to ease the production of datasets with labeled drifts were applied to Yelp and Airbnb datasets and tested using incremental classifiers respecting the stream mining paradigm to evaluate their ability to recover from the drifts.

Abstract

Systems and individuals produce data continuously. On the Internet, people share their knowledge, sentiments, and opinions, provide reviews about services and products, and so on. Automatically learning from these textual data can provide insights to organizations and institutions, thus preventing financial impacts, for example. To learn from textual data over time, the machine learning system must account for concept drift. Concept drift is a frequent phenomenon in real-world datasets and corresponds to changes in data distribution over time. For instance, a concept drift occurs when sentiments change or a word's meaning is adjusted over time. Although concept drift is frequent in real-world applications, benchmark datasets with labeled drifts are rare in the literature. To bridge this gap, this paper provides four textual drift generation methods to ease the production of datasets with labeled drifts. These methods were applied to Yelp and Airbnb datasets and tested using incremental classifiers respecting the stream mining paradigm to evaluate their ability to recover from the drifts. Results show that all methods have their performance degraded right after the drifts, and the incremental SVM is the fastest to run and recover the previous performance levels regarding accuracy and Macro F1-Score.

Methods for Generating Drift in Text Streams

TL;DR

Four textual drift generation methods to ease the production of datasets with labeled drifts were applied to Yelp and Airbnb datasets and tested using incremental classifiers respecting the stream mining paradigm to evaluate their ability to recover from the drifts.

Abstract

Systems and individuals produce data continuously. On the Internet, people share their knowledge, sentiments, and opinions, provide reviews about services and products, and so on. Automatically learning from these textual data can provide insights to organizations and institutions, thus preventing financial impacts, for example. To learn from textual data over time, the machine learning system must account for concept drift. Concept drift is a frequent phenomenon in real-world datasets and corresponds to changes in data distribution over time. For instance, a concept drift occurs when sentiments change or a word's meaning is adjusted over time. Although concept drift is frequent in real-world applications, benchmark datasets with labeled drifts are rare in the literature. To bridge this gap, this paper provides four textual drift generation methods to ease the production of datasets with labeled drifts. These methods were applied to Yelp and Airbnb datasets and tested using incremental classifiers respecting the stream mining paradigm to evaluate their ability to recover from the drifts. Results show that all methods have their performance degraded right after the drifts, and the incremental SVM is the fastest to run and recover the previous performance levels regarding accuracy and Macro F1-Score.
Paper Structure (23 sections, 5 figures, 1 table)

This paper contains 23 sections, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Text Stream Mining
  4. Concept Drift
  5. Text Drift Generation Methods
  6. Class-based methods
  7. Class Swap
  8. Class Shift
  9. Time-slice Removal
  10. Adjective Swap
  11. Experimental Results
  12. Experimental Protocol
  13. Datasets
  14. Airbnb
  15. Yelp
  16. ...and 8 more sections

Figures (5)

  • Figure 1: Representations of the Class Swap and Class Shift methods. Each color represents a class.
  • Figure 2: Representation of the Time Slice Removal method. The red squares represent randomly selected years to be removed from the text stream.
  • Figure 3: Representation of the Adjective Swap method. Orange rectangles represent the POS-tagged adjectives. The second row is the first one after applying the Adjective Swap method.
  • Figure 4: Class distribution of the datasets used in this paper.
  • Figure 5: Results for Airbnb (left) and Yelp (right). The dashed lines correspond to the drifts.