Interpretable Generative Models through Post-hoc Concept Bottlenecks

TL;DR

This work tackles the interpretability and scalability gap in generative models by introducing two post-hoc approaches, CB-AE and CC, that convert a pretrained generator into an inherently interpretable model without retraining from scratch or relying on abundant concept labels. The CB-AE inserts a bottleneck autoencoder between the generator halves, learning a concept space $c$ via an encoder $E$ and decoder $D$ while keeping $g_1$ and $g_2$ fixed; a lightweight CC provides concept predictions from $g_1(z)$ via a predictor $\Omega$. Interventions are enabled through optimization-based and cyclic losses, and a pseudo-label source $M$ (e.g., CLIP or supervised classifiers) drives concept alignment, which allows steering at test time and during training with minimal supervision. Across GANs and diffusion models on CelebA, CelebA-HQ, and CUB, CB-AE and CC achieve substantially higher steerability (avg ~25%) and are 4–15× faster to train than prior work, with large-scale human studies validating interpretability. The methods generalize to multiple model families and offer a practical path to scalable, post-hoc interpretable generation without expensive labeled data or scratch training.

Abstract

Concept bottleneck models (CBM) aim to produce inherently interpretable models that rely on human-understandable concepts for their predictions. However, existing approaches to design interpretable generative models based on CBMs are not yet efficient and scalable, as they require expensive generative model training from scratch as well as real images with labor-intensive concept supervision. To address these challenges, we present two novel and low-cost methods to build interpretable generative models through post-hoc techniques and we name our approaches: concept-bottleneck autoencoder (CB-AE) and concept controller (CC). Our proposed approaches enable efficient and scalable training without the need of real data and require only minimal to no concept supervision. Additionally, our methods generalize across modern generative model families including generative adversarial networks and diffusion models. We demonstrate the superior interpretability and steerability of our methods on numerous standard datasets like CelebA, CelebA-HQ, and CUB with large improvements (average ~25%) over the prior work, while being 4-15x faster to train. Finally, a large-scale user study is performed to validate the interpretability and steerability of our methods.

Figures (10)

• Figure 1: A. Prior work on interpretable generative models requires expensive generative model training from scratch. B. Our CB-AE and CC can be trained efficiently for post-hoc interpretability in a pretrained, frozen generative model $g_2\!\circ\! g_1$. C. Example concept intervention with CB-AE and corresponding concept vectors.
• Figure 2: Post-hoc CB-AE training for reconstruction, concept alignment, and intervention with a frozen pretrained generator $g_2\!\circ\! g_1$. Note that $\mathcal{L}_c$, $\mathcal{L}_i$ indicate cross-entropy loss and $\mathcal{L}_r$ indicates mean-squared-error loss. The darker lines indicate gradient flow during training.
• Figure 3: Samples generated using CB-AE with CelebA-HQ and CUB pretrained StyleGAN2 models along with concept probabilities.
• Figure 4: Concept intervention examples for CB-AE with CelebA-HQ pretrained StyleGAN2.
• Figure 5: Optimization-based concept intervention (opt-int) examples for CB-AE and CC with CelebA-HQ pretrained StyleGAN2.