Annotating Dimensions of Social Perception in Text: The First Sentence-Level Dataset of Warmth and Competence

TL;DR

W&C-Sent is the first sentence-level dataset annotating warmth (split into trust and sociability) and competence toward seven targets, built from SemEval stance data and the ABCDE corpus. The work argues that context matters for social trait inference and provides robust annotation procedures with reliability metrics, enabling nuanced NLP analyses of social perception. Comprehensive experiments show that even state-of-the-art LLMs struggle to reliably identify these traits, underscoring the need for high-quality, context-aware resources. The dataset and findings offer a valuable benchmark for discourse analysis, bias studies, and the development of socially aware NLP systems.

Abstract

Warmth (W) (often further broken down into Trust (T) and Sociability (S)) and Competence (C) are central dimensions along which people evaluate individuals and social groups (Fiske, 2018). While these constructs are well established in social psychology, they are only starting to get attention in NLP research through word-level lexicons, which do not completely capture their contextual expression in larger text units and discourse. In this work, we introduce Warmth and Competence Sentences (W&C-Sent), the first sentence-level dataset annotated for warmth and competence. The dataset includes over 1,600 English sentence--target pairs annotated along three dimensions: trust and sociability (components of warmth), and competence. The sentences in W&C-Sent are from social media and often express attitudes and opinions about specific individuals or social groups (the targets of our annotations). We describe the data collection, annotation, and quality-control procedures in detail, and evaluate a range of large language models (LLMs) on their ability to identify trust, sociability, and competence in text. W&C-Sent provides a new resource for analyzing warmth and competence in language and supports future research at the intersection of NLP and computational social science.

Figures (8)

• Figure 1: Examples showing divergences between dimensions of social perception.
• Figure 2: An example illustrating how the SemEval-2016 Stance dataset was used to extract sentence--target pairs for W&C-Sent. Specifically, a social media post is initially labeled as expressing a stance in favor of Donald Trump; however, because it also mentions Barack Obama, we additionally extract Barack Obama as a target and annotate it for competence, sociability, and trust.
• Figure 3: The distribution of median scores of our datasets' sentence--target pairs (i.e., 1,633 pairs for each dimension). This aggregation method filters out some noise and reduces the impact of outliers.
• Figure 4: Distribution of discretized mean-based labels, ordered from negative to positive.
• Figure 5: A figure showing the distribution of fine-grained unanimous judgments, across targets and dimensions. The figure shows how abundant the agreement is on sentences whose targets are Clinton and Trump in the trust dimension, and the complete absence of the target Environmentalists.