CommunityBench: Benchmarking Community-Level Alignment across Diverse Groups and Tasks

TL;DR

This paper introduces CommunityBench, a large-scale benchmark for community-level alignment that sits between universal and individualized approaches. Grounded in Common Identity and Common Bond (CICB) theory, it operationalizes four tasks—Preference Identification, Preference Distribution Prediction, Community-Consistent Generation, and Community Identification—using a Reddit-derived dataset spanning 6,919 communities (12,149 instances). A broad evaluation of 17 foundation models reveals that current LLMs struggle to capture nuanced community norms, especially for long-tail groups, but that richer community-contexts and training-based methods can facilitate downstream individual modeling by enabling individuals to be represented as intersections of multiple community identities. The work highlights both the promise and challenges of scalable, pluralistic alignment, suggesting that community-level signals can improve modeling of individual behavior while also identifying data-diversity and evaluation bottlenecks that warrant further research.

Abstract

Large language models (LLMs) alignment ensures model behaviors reflect human value. Existing alignment strategies primarily follow two paths: one assumes a universal value set for a unified goal (i.e., one-size-fits-all), while the other treats every individual as unique to customize models (i.e., individual-level). However, assuming a monolithic value space marginalizes minority norms, while tailoring individual models is prohibitively expensive. Recognizing that human society is organized into social clusters with high intra-group value alignment, we propose community-level alignment as a "middle ground". Practically, we introduce CommunityBench, the first large-scale benchmark for community-level alignment evaluation, featuring four tasks grounded in Common Identity and Common Bond theory. With CommunityBench, we conduct a comprehensive evaluation of various foundation models on CommunityBench, revealing that current LLMs exhibit limited capacity to model community-specific preferences. Furthermore, we investigate the potential of community-level alignment in facilitating individual modeling, providing a promising direction for scalable and pluralistic alignment.

Figures (11)

• Figure 1: Granularity of LLM alignment.One-size-fits-all (left) enforces universal values but may marginalize minority norms. Individual-level alignment (middle) offers personalization but faces data sparsity and implementation costs. Community-level alignment (right) bridges these extremes by capturing shared preference while preserving diversity.
• Figure 2: Community-level Alignment Tasks (left) and Benchmark Construction Pipeline (right). The left panel illustrates four core capabilities derived from Common Identity and Common Bond Theory (CICB): shared group identity, within-group heterogeneity, characteristic discourse practices, and behavioral traces, each motivating a corresponding task. The right panel shows how Reddit data are filtered, profiled, and transformed into query–response pairs with estimated opinion distributions for supervising these tasks.
• Figure 3: Dataset characteristics. (a) Preference entropy distribution, (b) Option length, (c) Query length, and (d) Community size statistics.
• Figure 4: Effect of profile granularity on community alignment. Comparison across Coarse, Summary, and Fine levels. Alignment performance consistently improves as granularity increases, indicating the value of richer contextual information.
• Figure 5: Long-tail distribution challenge in community alignment. We compare model accuracy against community size across various subreddits. Models achieve high accuracy on popular subreddits but struggle to represent niche cultures, highlighting the difficulty of aligning with the long tail of communities.