VASTU: Value-Aligned Social Toolkit for Online Content Curation

TL;DR

VASTU provides a standardized benchmark and evaluation framework for detecting community-valued online content, addressing fragmentation across feature-based, transformer, and language-model approaches. The study finds that community-specific models—particularly fully fine-tuned transformers like XLNet with local training and fine-tuned small language models—substantially outperform global models and prompting LLMs, underscoring the importance of learning local norms. Interpretability analyses reveal that signals driving value detection vary across communities, with SHAP highlighting prosociality and readability as context-dependent cues, while LLM explanations reflect broader normative cues. Practically, VASTU enables scalable, efficient, and community-grounded curation, suggesting modular, community-specific adapters over monolithic generalist models to surface what communities value while maintaining calibration and transparency.

Abstract

Detecting what content communities value is a foundational challenge for social computing systems -- from feed curation and content ranking to moderation tools and personalized recommendation systems. Yet existing approaches remain fragmented across methodological paradigms, and it remains unclear which methods best capture community-specific notions of value. We introduce VASTU (Value-Aligned Social Toolkit for Online Content Curation), a benchmark and evaluation framework for systematically comparing approaches to detecting community-valued content. VASTU includes a dataset of 75,000 comments from 15 diverse Reddit communities, annotated with community approval labels and rich linguistic features. Using VASTU, we evaluate feature-based models, transformers, prompted and fine-tuned language models under global versus community-specific training regimes. We find that community-specific models consistently outperform global approaches, with fine-tuned transformers achieving the strongest performance (0.72 AUROC). Notably, fine-tuned SLMs (0.65 AUROC) substantially outperform prompted LLMs (0.60 AUROC) despite being 100 times smaller. Counterintuitively, chain-of-thought prompting provides no benefit, and reasoning models perform the worst (0.53 AUROC), suggesting this task requires learning community norms rather than test-time reasoning. By releasing VASTU, we provide a standardized benchmark to advance research on value-aligned sociotechnical systems.

Figures (4)

• Figure 1: Feature set ablation for feature-based models. AUROC improves incrementally as feature sets are added, with LIWC providing the largest marginal gain.
• Figure 2: SHAP feature importance for XGBoost across topic categories. Cells show mean signed SHAP values aggregated across communities within each topic;. Feature importance varies substantially by community type.
• Figure 3: Error trade-offs across method families. While fine-tuned transformers and feature-based models tend to minimize False Negatives (prioritizing recall), vanilla LLMs show extreme conservatism in predicting valued content, resulting in high False Negative rates. Prompt engineering strategies and fine-tuning successfully shift LMs toward a more balanced error profile.
• Figure 4: Architecture of our transformer-based feature fusion model. The input Reddit comment is processed through a transformer encoder (BERT or XLNet) to obtain contextual representations, while linguistic features are extracted in parallel. Learnable weights allow the model to learn the optimal contribution of text embeddings versus linguistic features. The concatenated representation is passed through a classification head to predict whether the comment will be highly-valued by the community.