Measuring Value Expressions in Social Media Posts

TL;DR

This work tackles the challenge of measuring expressions of human values in social media by grounding the task in Schwartz’s 19 basic values and assembling a large, subjectively annotated corpus (N=1,079 annotators, 5,211 posts). It demonstrates substantial disagreement among humans and even between humans and off‑the‑shelf LLMs, motivating a perspectivist, personalization‑driven approach. A two‑stage modeling pipeline—finetuning a large language model on high‑consensus labels and then personalizing predictions with random forests using a Value Calibration Questionnaire—yields the strongest alignment with individual judgments, achieving a mean Spearman correlation of about $ ho\approx 0.334$ and a 66% improvement over the human–human baseline. The findings highlight the subjectivity inherent in value expression, offer a generalizable measurement framework adaptable to other value systems, and suggest practical implications for value‑aware feed design and auditing, while noting ethical considerations and limitations of platform generalization. The work advances methods for measuring human values in social media and provides a foundation for more nuanced, individualized value alignment in sociotechnical systems, with potential applications in fairer and more民主化 feed algorithms. $\rho$ and $MAE$ metrics illustrate the relative performance of humans and AI across consensus and individualized targets, underscoring the practical significance of personalization in value–expression modeling.

Abstract

The value alignment of sociotechnical systems has become a central debate but progress in this direction requires the measurement of the expressions of values. While the rise of large-language models offer new possible opportunities for measuring expressions of human values (e.g., humility or equality) in social media data, there remain both conceptual and practical challenges in operationalizing value expression in social media posts: what value system and operationalization is most applicable, and how do we actually measure them? In this paper, we draw on the Schwartz value system as a broadly encompassing and theoretically grounded set of basic human values, and introduce a framework for measuring Schwartz value expressions in social media posts at scale. We collect 32,370 ground truth value expression annotations from N=1,079 people on 5,211 social media posts representative of real users' feeds. We observe low levels of inter-rater agreement between people, and low agreement between human raters and LLM-based methods. Drawing on theories of interpretivism - that different people will have different subjective experiences of the same situation - we argue that value expression is (partially) in the eye of the beholder. In response, we construct a personalization architecture for classifying value expressions. We find that a system that explicitly models these differences yields predicted value expressions that people agree with more than they agree with other people. These results contribute new methods and understanding for the measurement of human values in social media data.

Figures (7)

• Figure 1: An example social media post for value annotation. All annotators in our dataset agree that this post strongly expresses Universal Concern, and they all agree that the post does not express Tradition---but, annotators disagree substantially on whether the post expresses Humility. In this paper, we model these interpersonal differences for more accurate value classification.
• Figure 2: The refined Schwartz value system. The system is hierarchically nested into a highest level: Outcomes for Self (right) vs Outcomes for Others (left), a high level: Self-Transcendence (top left), Conservation (bottom left), Self-Enhancement (bottom right) and Openness to Change (top left), and 19 low-level values. Adapted from schwartz2012refining
• Figure 3: Method for collecting and annotating tweets from user’s feeds
• Figure 4: Average Spearman rank correlation (across posts) comparing individual value annotations to 1) other individuals (gray, far left), 2) groups of increasing group size (light gray to dark gray, middle), 3) zero-shot GPT-4o (light red, right), 4) fine-tuned GPT-4o (red, right), and 5) personalized random forest (dark red, right).
• Figure A1: Top: Recursive tree structure for value annotation. Red indicates modifications we took to make the categories more interpretable for raters and salient to the social media context. Bottom: Example of recursive labeling scheme for gating post