Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Multilingual Social Media Insights: Aspect-based Comment Analysis

Longyin Zhang, Bowei Zou, Ai Ti Aw

TL;DR

This work tackles the noise and multilinguality of social media comments by introducing Comment Aspect Term Generation (CAT-G), a method that extracts central opinion targets to guide model attention. It combines supervised fine-tuning of multilingual large language models with Direct Preference Optimization (DPO) to align CAT outputs with human preferences, and introduces the first multilingual CAT-G dataset for EN, CN, MS, and ID. The CAT-G signals are integrated into monolingual and cross-lingual comment clustering (ComC), yielding measurable gains such as a cross-lingual clustering improvement of +$2.54$ $NMI$, demonstrating practical impact for robust social media analysis. The work also provides a benchmark framework, discusses limitations in distribution alignment and sentiment coupling, and addresses potential biases and risks associated with data-driven CAT generation.

Abstract

The inherent nature of social media posts, characterized by the freedom of language use with a disjointed array of diverse opinions and topics, poses significant challenges to downstream NLP tasks such as comment clustering, comment summarization, and social media opinion analysis. To address this, we propose a granular level of identifying and generating aspect terms from individual comments to guide model attention. Specifically, we leverage multilingual large language models with supervised fine-tuning for comment aspect term generation (CAT-G), further aligning the model's predictions with human expectations through DPO. We demonstrate the effectiveness of our method in enhancing the comprehension of social media discourse on two NLP tasks. Moreover, this paper contributes the first multilingual CAT-G test set on English, Chinese, Malay, and Bahasa Indonesian. As LLM capabilities vary among languages, this test set allows for a comparative analysis of performance across languages with varying levels of LLM proficiency.

Improving Multilingual Social Media Insights: Aspect-based Comment Analysis

TL;DR

This work tackles the noise and multilinguality of social media comments by introducing Comment Aspect Term Generation (CAT-G), a method that extracts central opinion targets to guide model attention. It combines supervised fine-tuning of multilingual large language models with Direct Preference Optimization (DPO) to align CAT outputs with human preferences, and introduces the first multilingual CAT-G dataset for EN, CN, MS, and ID. The CAT-G signals are integrated into monolingual and cross-lingual comment clustering (ComC), yielding measurable gains such as a cross-lingual clustering improvement of + , demonstrating practical impact for robust social media analysis. The work also provides a benchmark framework, discusses limitations in distribution alignment and sentiment coupling, and addresses potential biases and risks associated with data-driven CAT generation.

Abstract

The inherent nature of social media posts, characterized by the freedom of language use with a disjointed array of diverse opinions and topics, poses significant challenges to downstream NLP tasks such as comment clustering, comment summarization, and social media opinion analysis. To address this, we propose a granular level of identifying and generating aspect terms from individual comments to guide model attention. Specifically, we leverage multilingual large language models with supervised fine-tuning for comment aspect term generation (CAT-G), further aligning the model's predictions with human expectations through DPO. We demonstrate the effectiveness of our method in enhancing the comprehension of social media discourse on two NLP tasks. Moreover, this paper contributes the first multilingual CAT-G test set on English, Chinese, Malay, and Bahasa Indonesian. As LLM capabilities vary among languages, this test set allows for a comparative analysis of performance across languages with varying levels of LLM proficiency.

Paper Structure

This paper contains 22 sections, 2 equations, 4 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Resource & Approach
  3. Multilingual CAT Annotation
  4. CAT $\mathcal{G}$
  5. CAT $\mathcal{G}$-augmented ComC
  6. Experiments
  7. Results and Discussion
  8. CAT $\mathcal{G}$ results.
  9. Distribution of CAT $\mathcal{G}$.
  10. Case study.
  11. CAT $\mathcal{G}$ + ComC.
  12. Conclusion
  13. Limitations
  14. Potential Risks
  15. Acknowledgments
  16. ...and 7 more sections

Figures (4)

  • Figure 1: Multilingual social media comments suffer from heavy noise. We harness open-source LLMs to resolve the dilemma by extracting comment aspect terms, effectively acting as a form of noise cancellation, to enhance downstream social media analyses.
  • Figure 2: Distribution of Human&LLM labeled CATs.
  • Figure 3: Model training and validation.
  • Figure 4: Test data scale analysis.