Plug-and-Play Interpretable Responsible Text-to-Image Generation via Dual-Space Multi-facet Concept Control

TL;DR

This work addresses ethical concerns in text-to-image generation by enabling broad, scalable control over responsible content through a plug-and-play, dual-space framework. It introduces two add-on modules, RICE for embedding-space control and RIIDL for latent-space control, learned by distillation and enhanced with concept whitening to yield interpretable concept spaces conditioned on a user-specified set A_X. The dual-space integration allows balanced contributions from both pathways, enabling unified control over multiple responsible concepts without retraining the base model. Empirical results on WinoBias and I2P demonstrate competitive debiasing and safety while preserving image quality, and the approach scales to multiple concepts and new attributes with interpretable latent- and embedding-space modulations.

Abstract

Ethical issues around text-to-image (T2I) models demand a comprehensive control over the generative content. Existing techniques addressing these issues for responsible T2I models aim for the generated content to be fair and safe (non-violent/explicit). However, these methods remain bounded to handling the facets of responsibility concepts individually, while also lacking in interpretability. Moreover, they often require alteration to the original model, which compromises the model performance. In this work, we propose a unique technique to enable responsible T2I generation by simultaneously accounting for an extensive range of concepts for fair and safe content generation in a scalable manner. The key idea is to distill the target T2I pipeline with an external plug-and-play mechanism that learns an interpretable composite responsible space for the desired concepts, conditioned on the target T2I pipeline. We use knowledge distillation and concept whitening to enable this. At inference, the learned space is utilized to modulate the generative content. A typical T2I pipeline presents two plug-in points for our approach, namely; the text embedding space and the diffusion model latent space. We develop modules for both points and show the effectiveness of our approach with a range of strong results.

Figures (6)

• Figure 1: Our unique plug-and-play interpretable approach simultaneously controls a range of concepts for responsible and fair image generation with text-to-image pipelines. Our method enables control over both text embedding space and latent diffusion space. Shown examples compare unfair and unsafe generation for the given prompts by the Stable Diffusion (top), along with their responsible counterparts resulting from our approach influencing the text encoder (middle) and the diffusion model (bottom). We control individual concepts for diverse/safe generation in these examples while modeling them in continuous composite responsible semantic spaces.
• Figure 2: (a) Illustration of responsible concept space learned by the RICE module, aiming to generate $z_{\text{clip}}^{\text{final}}$ embeddings along the responsible aspects in $\mathcal{A}_{\text{X}}$. Interconnected fairness related subspaces for race, gender, and age are illustrated. (b) Example outputs of RICE controlled T2I process, demonstrating a range of modulation along the concepts, which is possible due to the multi-facet control enabled by the RICE embedding space. (c) Depiction of the distillation process used for the RIIDL module, where the student (RIIDL) learns from the teacher (Diffusion Model) over $\tau$ time-steps. At each step, the loss - see Eq. (\ref{['eq:diff']}) - aligns the RIIDL module with the aspects of $\mathcal{A}_{X}$. (d) At inference, textual embeddings get fed into RIIDL. We illustrate control over $\mathcal{A}_{\text{age}}$. The latent vectors from RIIDL are injected into the T2I Diffusion Model noisy latents. We explore this injection at varying time-step (differentiated vertically in the figure). The best control is observed in the early stages $(\tau = 0~\text{to}~30\%)$. Additional examples provided in the supplementary material. (e) Our method allows a scalable control over all concepts covered in $\mathcal{A}_X$ in a composite manner.
• Figure 3: Pair-wise comparison for the Stable Diffusion (SD) baseline and our plugin to it for responsible generation along age, gender and race. SD images contain stereotypical associations to the profession, which get removed by our method by often accounting for more than one responsible attributes, while maintaining high image quality.
• Figure 4: Controlled composite responsible generation using the proposed method. By using different concepts in $\mathcal{A}_X$ in Eq. (\ref{['eq:responsible']}), and employing dual space control in (\ref{['eq:dualspace']}), our technique can enable responsibility along single or multiple concepts, as desired. Provided the prompt “A doctor” to Stable Diffusion, alignment is achieved for a target concept composition (top label) for the attributes noted at the bottom of each image set. See supplementary material for more examples.
• Figure 5: Examples of extending $\mathcal{A}_{\text{resp}}$ with additional attributes - possibly unrelated to core responsible concepts. Extensions with smile and glasses are shown. In addition to varying images along responsible concepts of age, race and gender, our method is able to seamlessly incorporate the additional concepts in images.