Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Concept Lens: Visually Analyzing the Consistency of Semantic Manipulation in GANs

Sangwon Jeong, Mingwei Li, Matthew Berger, Shusen Liu

TL;DR

This work tackles understanding how semantic manipulations in GAN latent spaces behave across diverse regions of the latent space. It introduces Concept Lens, a bi-hierarchical visual analytics design that jointly analyzes large sets of concepts and codes, using hierarchical encoding, Cartesian product views, and a color-driven consistency metric to reveal where concept directions yield stable edits. Key contributions include a formalized notion of concept-distance and edit-consistency, a reclustering capability for localized analysis, and demonstrations across multiple domains using StyleGAN2 with SEFA. The approach provides a scalable, interpretable framework for diagnosing concept discovery methods and improving semantic manipulation in generative models, with potential applicability to diffusion models and beyond.

Abstract

As applications of generative AI become mainstream, it is important to understand what generative models are capable of producing, and the extent to which one can predictably control their outputs. In this paper, we propose a visualization design, named Concept Lens, for jointly navigating the data distribution of a generative model, and concept manipulations supported by the model. Our work is focused on modern vision-based generative adversarial networks (GAN), and their learned latent spaces, wherein concept discovery has gained significant interest as a means of image manipulation. Concept Lens is designed to support users in understanding the diversity of a provided set of concepts, the relationship between concepts, and the suitability of concepts to give semantic controls for image generation. Key to our approach is the hierarchical grouping of concepts, generated images, and the associated joint exploration. We show how Concept Lens can reveal consistent semantic manipulations for editing images, while also serving as a diagnostic tool for studying the limitations and trade-offs of concept discovery methods.

Concept Lens: Visually Analyzing the Consistency of Semantic Manipulation in GANs

TL;DR

This work tackles understanding how semantic manipulations in GAN latent spaces behave across diverse regions of the latent space. It introduces Concept Lens, a bi-hierarchical visual analytics design that jointly analyzes large sets of concepts and codes, using hierarchical encoding, Cartesian product views, and a color-driven consistency metric to reveal where concept directions yield stable edits. Key contributions include a formalized notion of concept-distance and edit-consistency, a reclustering capability for localized analysis, and demonstrations across multiple domains using StyleGAN2 with SEFA. The approach provides a scalable, interpretable framework for diagnosing concept discovery methods and improving semantic manipulation in generative models, with potential applicability to diffusion models and beyond.

Abstract

As applications of generative AI become mainstream, it is important to understand what generative models are capable of producing, and the extent to which one can predictably control their outputs. In this paper, we propose a visualization design, named Concept Lens, for jointly navigating the data distribution of a generative model, and concept manipulations supported by the model. Our work is focused on modern vision-based generative adversarial networks (GAN), and their learned latent spaces, wherein concept discovery has gained significant interest as a means of image manipulation. Concept Lens is designed to support users in understanding the diversity of a provided set of concepts, the relationship between concepts, and the suitability of concepts to give semantic controls for image generation. Key to our approach is the hierarchical grouping of concepts, generated images, and the associated joint exploration. We show how Concept Lens can reveal consistent semantic manipulations for editing images, while also serving as a diagnostic tool for studying the limitations and trade-offs of concept discovery methods.
Paper Structure (7 sections, 3 equations, 5 figures)

This paper contains 7 sections, 3 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Related Works
  3. Design Requirements
  4. Concept Lens Design
  5. Use Cases
  6. Discussion
  7. Experimental details

Figures (5)

  • Figure 1: The selected concept applies "closed eyes" to most codes, while it doesn't to the top selection in the code hierarchy. The top group is kittens, and, in our dataset, we find that kittens usually do not close their eyes. For the cats in the bottom group, "closed eyes" concept is combined with "mouth open".
  • Figure 2: Re-clustering the concept hierarchy according to a set of closely related codes helps reveal the more consistent and high-quality concept edit. Re-clustering can be performed by brushing a subset of code in the code hierarchy (1) and clicking on the recluster button (2). Re-clustered directions are pointed by the arrows.
  • Figure :
  • Figure :
  • Figure :