Density-Guided Response Optimization: Community-Grounded Alignment via Implicit Acceptance Signals

TL;DR

DGRO is position as a practical alignment alternative for communities where explicit preference supervision is unavailable or misaligned with situated practices, and the implications and risks of learning from emergent acceptance behavior are discussed.

Abstract

Language models deployed in online communities must adapt to norms that vary across social, cultural, and domain-specific contexts. Prior alignment approaches rely on explicit preference supervision or predefined principles, which are effective for well-resourced settings but exclude most online communities -- particularly those without institutional backing, annotation infrastructure, or organized around sensitive topics -- where preference elicitation is costly, ethically fraught, or culturally misaligned. We observe that communities already express preferences implicitly through what content they accept, engage with, and allow to persist. We show that this acceptance behavior induces measurable geometric structure in representation space: accepted responses occupy coherent, high-density regions that reflect community-specific norms, while rejected content falls in sparser or misaligned areas. We operationalize this structure as an implicit preference signal for alignment and introduce density-guided response optimization (DGRO), a method that aligns language models to community norms without requiring explicit preference labels. Using labeled preference data, we demonstrate that local density recovers pairwise community judgments, indicating that geometric structure encodes meaningful preference signal. We then apply DGRO in annotation-scarce settings across diverse communities spanning platform, topic, and language. DGRO-aligned models consistently produce responses preferred by human annotators, domain experts, and model-based judges over supervised and prompt-based baselines. We position DGRO as a practical alignment alternative for communities where explicit preference supervision is unavailable or misaligned with situated practices, and discuss the implications and risks of learning from emergent acceptance behavior.

Figures (6)

• Figure 1: Conceptual Representation of the Community Consensus Surface. The Z-axis represents a normative log-density, reflecting the implicit filtering of responses by community standards through moderation and collective feedback lampe2004slashchandrasekharan2017you. High-density regions correspond to a coherent, low-dimensional manifold of accepted responses in representation space arora2018linearli2020sentence. The separation between preferred ($r^+$) and non-preferred ($r^-$) responses across this surface reflects an acceptance--preference correspondence, motivating preference learning and alignment without explicit annotation christiano2017deepouyang2022trainingrafailov2023direct.
• Figure 2: Relative accuracy of DRGO-aligned models expressed as a percentage of baseline DPO performance, computed as $100 \times (\text{DRGO} / \text{baseline})$, where $100\%$ denotes parity with the baseline. Error bars denote $\pm 1$ standard error estimated via bootstrap resampling ($n{=}500$), with uncertainty propagated using a first-order delta method.
• Figure 3: Local accuracy saturates quickly with neighborhood size. Shown is the worst-case absolute accuracy loss across communities relative to each community’s best-performing neighborhood size, demonstrating that performance remains within a few percentage points of optimal across a wide range of $k$.
• Figure 4: Accuracy across communities. Bars show kNN, global density, and local density baselines, evaluated on the easy subset of examples. Vertical ticks denote supervised reward model (RM) accuracy. The dashed vertical line at $0.50$ marks random-chance performance.
• Figure 5: Higher human agreement correlates with higher local accuracy. Each point is an agreement-strength bin from a subreddit. The moderately strong positive correlation ($\rho_s = 0.48$, $p < 10^{-4}$) suggests that judge accuracy improves in regions where community preferences are more clearly differentiated. The fitted line is shown for visualization only; significance is assessed with rank correlations.