From Millions of Tweets to Actionable Insights: Leveraging LLMs for User Profiling

TL;DR

The paper tackles scalable, transferable, and interpretable user profiling on social media by introducing a two-stage LLM-based framework that leverages domain-defining statements and semi-supervised filtering to distill user history into concise, interpretable profiles. It builds a Persian political knowledge base and a large Persian Twitter dataset (PersianPol6M), then generates both abstractive profiles (concise natural-language summaries) and extractive profiles (representative tweets) with minimal labeled data. An LLM-assisted evaluation framework with human validation demonstrates that the proposed approach outperforms state-of-the-art LLM-based and traditional profiling methods by 9.8%, with robust performance across downstream tasks. The work also releases PersianPol6M and demonstrates adaptability across domains, reducing labeling needs while maintaining interpretability and applicability for downstream social-network analyses.

Abstract

Social media user profiling through content analysis is crucial for tasks like misinformation detection, engagement prediction, hate speech monitoring, and user behavior modeling. However, existing profiling techniques, including tweet summarization, attribute-based profiling, and latent representation learning, face significant limitations: they often lack transferability, produce non-interpretable features, require large labeled datasets, or rely on rigid predefined categories that limit adaptability. We introduce a novel large language model (LLM)-based approach that leverages domain-defining statements, which serve as key characteristics outlining the important pillars of a domain as foundations for profiling. Our two-stage method first employs semi-supervised filtering with a domain-specific knowledge base, then generates both abstractive (synthesized descriptions) and extractive (representative tweet selections) user profiles. By harnessing LLMs' inherent knowledge with minimal human validation, our approach is adaptable across domains while reducing the need for large labeled datasets. Our method generates interpretable natural language user profiles, condensing extensive user data into a scale that unlocks LLMs' reasoning and knowledge capabilities for downstream social network tasks. We contribute a Persian political Twitter (X) dataset and an LLM-based evaluation framework with human validation. Experimental results show our method significantly outperforms state-of-the-art LLM-based and traditional methods by 9.8%, demonstrating its effectiveness in creating flexible, adaptable, and interpretable user profiles.

Figures (10)

• Figure 1: Dataset curation pipeline for political content related to the 2024 Iranian presidential election, including data collection, knowledge base construction, semi-supervised labeling, and BERT-based classification
• Figure 2: Venn diagram illustrating the overlap of unique tweets among the three main candidates in the 2024 Iranian Presidential Election: Pezeshkian, Jalili and Ghalibaf
• Figure 3: Confusion matrices for: (a) unsupervised labeling on human annotations, (b) classifier fine-tuning with semi-supervised data on test set, and (c) final classifier evaluation using borderline data annotated by human
• Figure 4: Histogram of mean distances between tweets and their top 10 closest knowledge base chunks. Lower distances indicate stronger alignment with political content
• Figure 5: Step-by-step illustration of the user profiling and evaluation process. The pipeline includes (1) user splitting, (2) domain specific defining statement generation, (3) heuristic tweet selection to get a pool of tweets for annotation and profiling, (4) user profiling with tweet pool and statements (5) statement-user pairs ground truth generation by human using pool of tweets, (6) LLM-based evaluation on user-statement pairs with user profiles as context, and (7) profile comparison based answers with human ground truth