RespoDiff: Dual-Module Bottleneck Transformation for Responsible & Faithful T2I Generation

TL;DR

RespoDiff introduces a dual-module bottleneck transformation for diffusion-based text-to-image generation to achieve responsible (fair/safe) outputs while preserving semantic fidelity and image quality. It learns a Responsible Concept Alignment Module (RAM) to steer neutral prompts toward target concepts and a Semantic Alignment Module (SAM) to maintain fidelity to the original diffusion trajectory, coordinated by a score-matching objective. The method demonstrates substantial improvements in fairness and safety on Winobias and I2P benchmarks, generalizes to unseen prompts, and integrates with SDXL without sacrificing fidelity. Its modular, plug-and-play design supports extension to transformer-based backbones and LoRA-like comparisons, offering a scalable path to safer AI-driven image synthesis.

Abstract

The rapid advancement of diffusion models has enabled high-fidelity and semantically rich text-to-image generation; however, ensuring fairness and safety remains an open challenge. Existing methods typically improve fairness and safety at the expense of semantic fidelity and image quality. In this work, we propose RespoDiff, a novel framework for responsible text-to-image generation that incorporates a dual-module transformation on the intermediate bottleneck representations of diffusion models. Our approach introduces two distinct learnable modules: one focused on capturing and enforcing responsible concepts, such as fairness and safety, and the other dedicated to maintaining semantic alignment with neutral prompts. To facilitate the dual learning process, we introduce a novel score-matching objective that enables effective coordination between the modules. Our method outperforms state-of-the-art methods in responsible generation by ensuring semantic alignment while optimizing both objectives without compromising image fidelity. Our approach improves responsible and semantically coherent generation by 20% across diverse, unseen prompts. Moreover, it integrates seamlessly into large-scale models like SDXL, enhancing fairness and safety. Code will be released upon acceptance.

Figures (12)

• Figure 1: Illustration of RespoDiff : RespoDiff performs reverse diffusion to timestep $t$ using the prompt "a person", obtaining latent $\bm{z}_{t, \text{neu}}$ as the neutral denoised latent for all forward processes. Forward diffusion with the RAM and "a person" predicts a neutral score, with mean-squared error between neutral and target scores updating the RAM. To maintain faithfulness to the original diffusion process, forward diffusion with both RAM and SAM generates a neutral score, with mean-squared error between neutral and original scores updating the SAM.
• Figure 2: Comparison of RespoDiff and SD in generating profession images by gender (top: women in first 4 columns, men in the rest) and race (bottom: Black in first 2, Asian in next 3, White in last 2). RespoDiff better reflects target attributes while maintaining fidelity to SD outputs.
• Figure 3: Qualitative comparison of safe generation. RespoDiff removes nudity and violence present in SD outputs, producing safer and more appropriate images.
• Figure 4: Comparison of gender fairness, alignment and quality metrics with SDXL.
• Figure 5: Inference with RespoDiff. Stable Diffusion (top) produces biased generations for the prompt “a doctor”, predominantly depicting male doctors. RespoDiff (bottom) applies concept-specific RAM and SAM modules at inference, sampling across gender with equal probability, and yields balanced outputs while preserving semantic fidelity to the prompt.