Compositional Bias Control in Large Language Models: Preference Learning Fails, Supervision Succeeds

TL;DR

The paper addresses the challenge of composing fairness constraints in large language models by evaluating six control strategies on a compositional bias task requiring agentic and communal descriptors. It finds that explicit supervision via supervised fine-tuning (SFT) achieves near-perfect conformance with high lexical diversity and reasonable fluency, while hard symbolic constraints (Ctrl-G AND) guarantee compliance but suppress diversity. Direct Preference Optimization (DPO) fails to enforce conjunctions, revealing a fundamental limitation of preference-based alignment for compositional constraints. Iterative Nullspace Projection (INLP) over-corrects and erases meaningful content, and decoding-time hard constraints with OR can yield high fluency but no true compositional balance. The study concludes that explicit positive supervision is necessary to mitigate compositional biases effectively, highlighting a critical difference between preference learning and supervision for fair and fluent controlled generation, and it provides a reproducible benchmark for future work.

Abstract

Large Language Models (LLMs) still produce gender-stereotyped language even in occupation-neutral contexts that reflect deep societal biases (Rudinger et al., 2018). To address this, prior work has proposed prompting, constrained decoding (Dathathri et al., 2020; Zhou et al., 2024), post-processing, and fine-tuning-based alignment (Rafailov et al., 2023; Ravfogel et al., 2022). However, the comparative efficacy and learning dynamics remain little understood. We report a comparative analysis of six control techniques for bias mitigation: prompt-only, generate-and-filter, DFA-based Ctrl-G decoding, Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Iterative Nullspace Projection (INLP). We evaluate each method on a compositional constraint task. This task requires generating sentences that contain at least one agentic and one communal descriptor for each of the twenty Winogender-derived occupations. We quantify trade-offs between control strength and naturalness with evaluations of constraint compliance, lexical diversity, and fluency. Our results reveal key contrasts among the methods: SFT achieves 99.87 +- 0.15% compliance and high lexical diversity, while DPO, despite similar training stability, fails at 4.53 +- 0.82%. Ctrl-G guarantees perfect compliance, but at the cost of severely reduced fluency and diversity. Preference-based learning fundamentally differs: it cannot satisfy compositional constraints, as binary preference signals encode ranking, not logical conjunctions. Only explicit positive supervision enables mitigation of compositional biases; preference-based alignment fails to generalize logical structures, underscoring the limitations of preference learning and the necessity of explicit supervision for fair and fluent controlled generation.

Figures (4)

• Figure 1: Constraint compliance across methods.
• Figure 2: Diversity metrics across methods.
• Figure 3: Cross-seed variance across methods.
• Figure 4: Trade-off between constraint strength and fluency.