CHOIR: Collaborative Harmonization fOr Inference Robustness

TL;DR

The paper tackles the sensitivity of large language models to demographic cues embedded in persona prompts, which can cause divergent reasoning paths. It introduces CHOIR, a no-training, inference-time framework that constructs multiple counterfactual personas and harmonizes their logits through dynamic, consensus-based weighting to produce more robust answers. Empirical results show up to 26.4% improvement for individual demographics and 19.2% average gain across five demographics, with strong generalization across models, tasks, and prompt templates, and compatibility with other ensemble methods. This approach reframes demographic variation as a constructive signal, enabling more reliable and socially aware LLM reasoning without additional training. The work highlights practical implications for safer, fairer, and more dependable AI reasoning in real-world applications.

Abstract

Persona-assigned Large Language Models (LLMs) can adopt diverse roles, enabling personalized and context-aware reasoning. However, even minor demographic perturbations in personas, such as simple pronoun changes, can alter reasoning trajectories, leading to divergent sets of correct answers. Instead of treating these variations as biases to be mitigated, we explore their potential as a constructive resource to improve reasoning robustness. We propose CHOIR (Collaborative Harmonization fOr Inference Robustness), a test-time framework that harmonizes multiple persona-conditioned reasoning signals into a unified prediction. CHOIR orchestrates a collaborative decoding process among counterfactual personas, dynamically balancing agreement and divergence in their reasoning paths. Experiments on various reasoning benchmarks demonstrate that CHOIR consistently enhances performance across demographics, model architectures, scales, and tasks - without additional training. Improvements reach up to 26.4% for individual demographic groups and 19.2% on average across five demographics. It remains effective even when base personas are suboptimal. By reframing persona variation as a constructive signal, CHOIR provides a scalable and generalizable approach to more reliable LLM reasoning.

Figures (6)

• Figure 1: LLM performance is sensitive to simple pronoun perturbations. This figure shows accuracy variations of the Llama-1B model on GSM8K across five different base personas, each perturbed with he, she, and they pronouns. Even when overall accuracies are similar (pairs i, ii, iii), the sets of correctly solved problems differ (Figure \ref{['fig:two']}). Personas for $G_{\text{Gender}}^{(i)}$ are listed in Table \ref{['tab:gender_group']}; detailed analysis is in §\ref{['ssec:perturbation']}.
• Figure 2: Similar overall accuracy reveals complementary reasoning strengths. This figure shows the overlap in correct answers for three pairs of personas from Figure \ref{['fig:one']} with nearly identical accuracy. For example, in (i), the he and she personas differ by only 0.1% in accuracy yet share 315 correct answers, with 73 and 74 uniquely solved, respectively. These non-overlapping sets demonstrate that minimal perturbations can lead to distinct reasoning paths (See §\ref{['ssec:accuracy']} for details.)
• Figure 3: Performance of CHOIR vs. PersonaMajority across four reasoning datasets. Bars represent average accuracy across five demographic attributes for Llama and Qwen models.
• Figure 4: Analysis of the pre-trained knowledge weight $\lambda^{(0)}_t$ on GSM8K (Gender). Results reveal a model-specific optimal balance, indicating that different architectures have varying reliance on their base priors.
• Figure 5: CHOIR + Self-Consistency (SC) vs. SC baseline on GSM8K and CommonsenseQA (Llama-8B, Gender). Integrating SC reasoning with CHOIR improves accuracy across both datasets.