How Bias Binds: Measuring Hidden Associations for Bias Control in Text-to-Image Compositions

TL;DR

This work tackles bias in text-to-image generation arising from semantic bindings within compositional prompts. It introduces a Bias Adherence Score (BA-Score) to quantify how object-attribute bindings influence bias, and a training-free Context Bias Control (CBC) framework that decouples sensitive attribute components from main-object semantics and injects residuals to steer generation. CBC leverages Schmidt orthogonalization to create attribute-orthogonal embeddings, latent-space BA-Score guidance, and attention rescaling to maintain stability, achieving over 10% debiasing gains in compositional generation without sacrificing image quality. Experiments on the Winobias benchmark demonstrate that semantic-context bindings can amplify bias under existing methods, and the proposed CBC approach yields improved fairness metrics (e.g., FD, VQA, AFS) while preserving text-image alignment. The study highlights limitations of current debiasing strategies for semantically bound contexts and points toward training-free, composition-aware debiasing as a practical path for robust, fair T2I systems.

Abstract

Text-to-image generative models often exhibit bias related to sensitive attributes. However, current research tends to focus narrowly on single-object prompts with limited contextual diversity. In reality, each object or attribute within a prompt can contribute to bias. For example, the prompt "an assistant wearing a pink hat" may reflect female-inclined biases associated with a pink hat. The neglected joint effects of the semantic binding in the prompts cause significant failures in current debiasing approaches. This work initiates a preliminary investigation on how bias manifests under semantic binding, where contextual associations between objects and attributes influence generative outcomes. We demonstrate that the underlying bias distribution can be amplified based on these associations. Therefore, we introduce a bias adherence score that quantifies how specific object-attribute bindings activate bias. To delve deeper, we develop a training-free context-bias control framework to explore how token decoupling can facilitate the debiasing of semantic bindings. This framework achieves over 10% debiasing improvement in compositional generation tasks. Our analysis of bias scores across various attribute-object bindings and token decorrelation highlights a fundamental challenge: reducing bias without disrupting essential semantic relationships. These findings expose critical limitations in current debiasing approaches when applied to semantically bound contexts, underscoring the need to reassess prevailing bias mitigation strategies.

Figures (6)

• Figure 1: Conceptual illustration of the association bias in compositional text-to-image generation. Orange and blue boxes denote the gender bias. Current debiasing frameworks neglect the hidden bias stemming from semantic bindings, such as pink hat or blond hair, causing low-quality generation or persistent biases after debiasing.
• Figure 2: Text correlation to male or female prototype embeddings extracted via CLIP encoder.
• Figure 3: Our proposed CBC framework that decouples the token from sensitive attributes, leverages the BA-Score to measure bias tendency, and adjusts the token with sensitive attribute residuals for unbiased generation.
• Figure 4: Generated images using different debiasing approaches.
• Figure 5: Bias results (FD) in binding conditions with (a) objects and (b) color attributes. The corresponding averaged results for objects and attributes are presented in (c) and (d). Results from other professions are reported in the supplementary.