EmoXpt: Analyzing Emotional Variances in Human Comments and LLM-Generated Responses

TL;DR

EmoXpt tackles the problem of understanding emotional dynamics around generative AI by jointly analyzing human sentiments toward AI and the sentiment embedded in ChatGPT's replies. The method uses unsupervised K-means clustering on 768-dimensional BERT-based word and sentence embeddings, applied to a Twitter-derived dataset of human tweets, ChatGPT responses, and user comments, with evaluation via Silhouette scores and t-SNE visualizations. Key findings show that human comments are predominantly negative while ChatGPT responses skew strongly positive, indicating a positivity bias in AI-generated text; the approach also reveals differences in clustering quality between word- and sentence-level representations. The work advances sentiment analysis by comparing human and AI-generated language, informing the design of emotionally intelligent conversational agents and the interpretation of public discourse on generative AI. Future work aims to broaden data sources, languages, and model comparisons to capture a richer spectrum of emotional expression and longitudinal trends.

Abstract

The widespread adoption of generative AI has generated diverse opinions, with individuals expressing both support and criticism of its applications. This study investigates the emotional dynamics surrounding generative AI by analyzing human tweets referencing terms such as ChatGPT, OpenAI, Copilot, and LLMs. To further understand the emotional intelligence of ChatGPT, we examine its responses to selected tweets, highlighting differences in sentiment between human comments and LLM-generated responses. We introduce EmoXpt, a sentiment analysis framework designed to assess both human perspectives on generative AI and the sentiment embedded in ChatGPT's responses. Unlike prior studies that focus exclusively on human sentiment, EmoXpt uniquely evaluates the emotional expression of ChatGPT. Experimental results demonstrate that LLM-generated responses are notably more efficient, cohesive, and consistently positive than human responses.

Figures (10)

• Figure 1: The Overview of EmoXpt
• Figure 2: Percentage of Tweets Collected per Hashtag
• Figure 3: Topic Analysis of Human Comments
• Figure 4: Word Frequency of ChatGPT Responses.
• Figure 5: Cleaning Steps For Human Comments.