Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Transit Pulse: Utilizing Social Media as a Source for Customer Feedback and Information Extraction with Large Language Model

Jiahao Wang, Amer Shalaby

TL;DR

This work proposes a novel approach to extracting and analyzing transit-related information, including sentiment and sarcasm detection, identification of unusual system problems, and location data from social media, and employs Large Language Models (LLM), specifically Llama 3, for a streamlined analysis free from pre-established topic labels.

Abstract

Users of the transit system flood social networks daily with messages that contain valuable insights crucial for improving service quality. These posts help transit agencies quickly identify emerging issues. Parsing topics and sentiments is key to gaining comprehensive insights to foster service excellence. However, the volume of messages makes manual analysis impractical, and standard NLP techniques like Term Frequency-Inverse Document Frequency (TF-IDF) fall short in nuanced interpretation. Traditional sentiment analysis separates topics and sentiments before integrating them, often missing the interaction between them. This incremental approach complicates classification and reduces analytical productivity. To address these challenges, we propose a novel approach to extracting and analyzing transit-related information, including sentiment and sarcasm detection, identification of unusual system problems, and location data from social media. Our method employs Large Language Models (LLM), specifically Llama 3, for a streamlined analysis free from pre-established topic labels. To enhance the model's domain-specific knowledge, we utilize Retrieval-Augmented Generation (RAG), integrating external knowledge sources into the information extraction pipeline. We validated our method through extensive experiments comparing its performance with traditional NLP approaches on user tweet data from the real world transit system. Our results demonstrate the potential of LLMs to transform social media data analysis in the public transit domain, providing actionable insights and enhancing transit agencies' responsiveness by extracting a broader range of information.

Transit Pulse: Utilizing Social Media as a Source for Customer Feedback and Information Extraction with Large Language Model

TL;DR

This work proposes a novel approach to extracting and analyzing transit-related information, including sentiment and sarcasm detection, identification of unusual system problems, and location data from social media, and employs Large Language Models (LLM), specifically Llama 3, for a streamlined analysis free from pre-established topic labels.

Abstract

Users of the transit system flood social networks daily with messages that contain valuable insights crucial for improving service quality. These posts help transit agencies quickly identify emerging issues. Parsing topics and sentiments is key to gaining comprehensive insights to foster service excellence. However, the volume of messages makes manual analysis impractical, and standard NLP techniques like Term Frequency-Inverse Document Frequency (TF-IDF) fall short in nuanced interpretation. Traditional sentiment analysis separates topics and sentiments before integrating them, often missing the interaction between them. This incremental approach complicates classification and reduces analytical productivity. To address these challenges, we propose a novel approach to extracting and analyzing transit-related information, including sentiment and sarcasm detection, identification of unusual system problems, and location data from social media. Our method employs Large Language Models (LLM), specifically Llama 3, for a streamlined analysis free from pre-established topic labels. To enhance the model's domain-specific knowledge, we utilize Retrieval-Augmented Generation (RAG), integrating external knowledge sources into the information extraction pipeline. We validated our method through extensive experiments comparing its performance with traditional NLP approaches on user tweet data from the real world transit system. Our results demonstrate the potential of LLMs to transform social media data analysis in the public transit domain, providing actionable insights and enhancing transit agencies' responsiveness by extracting a broader range of information.

Paper Structure

This paper contains 22 sections, 21 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Manual Identification
  4. Lexicon-Based Approaches
  5. Machine Learning-Based Approaches
  6. Large Language Model-Based Approaches
  7. Methodology
  8. Traditional Classification Task
  9. TF-IDF
  10. BERT Tokenizer and BERT Model
  11. Information Extraction
  12. Retrieval Augmented Generation (RAG) System
  13. Experiments
  14. Datasets
  15. Models and Environment Setup
  16. ...and 7 more sections

Figures (21)

  • Figure 1: Framework for the Traditional Classification Task, illustrating the steps from data processing to model evaluation.
  • Figure 2: Information Extraction Pipeline with Llama 3, demonstrating the process of embedding tweets into structured prompts and aggregating the extracted information.
  • Figure 3: Examples of Station Information Misinterpretation in Tweets.
  • Figure 4: Retrieval Augmented Generation (RAG) System, showing the process of embedding external knowledge and matching it with user queries to enhance information extraction accuracy.
  • Figure 5: System map of the Toronto Transit Commission (TTC) TTC_map.
  • ...and 16 more figures