Counterfactual Image Editing

TL;DR

This paper formalizes the counterfactual image editing task using formal language, modeling the causal relationships between latent generative factors and images through a special type of model called augmented structural causal models (ASCMs).

Abstract

Counterfactual image editing is an important task in generative AI, which asks how an image would look if certain features were different. The current literature on the topic focuses primarily on changing individual features while remaining silent about the causal relationships between these features, as present in the real world. In this paper, we formalize the counterfactual image editing task using formal language, modeling the causal relationships between latent generative factors and images through a special type of model called augmented structural causal models (ASCMs). Second, we show two fundamental impossibility results: (1) counterfactual editing is impossible from i.i.d. image samples and their corresponding labels alone; (2) even when the causal relationships between the latent generative factors and images are available, no guarantees regarding the output of the model can be provided. Third, we propose a relaxation for this challenging problem by approximating non-identifiable counterfactual distributions with a new family of counterfactual-consistent estimators. This family exhibits the desirable property of preserving features that the user cares about across both factual and counterfactual worlds. Finally, we develop an efficient algorithm to generate counterfactual images by leveraging neural causal models.

Figures (17)

• Figure 1: (Left) A causal graph depicting the causal relationships among features. (Right) Image editing results are displayed, with the first row showing edits incorporating causal relations, and the second row without them. Each column represents a unique counterfactual query, altering the age, gender, and gray hair of the individuals. These instances provide preliminary evidence that the causal approach introduced in this paper ensures the preservation of the relevant causal invariances for the query across both factual and counterfactual images.
• Figure 2: $P(\mathbf{V})$ induced by the ASCM in Example. \ref{['ex:face-ascm']}.
• Figure 3: (a) The proxy generator $\widehat{\mathcal{M}}$ is compatible with the same observational distributions with the unobserved true model but is not guaranteed to induce the same image counterfactual distributions. (b) Two different image counterfactual distributions in \ref{['ex:face-collapse']}. Each sample from a $P(\mathbf{I}, \mathbf{I}_{Y=0})$ has an initial image $\mathbf{i}$ and a counterfactual image $\mathbf{i}'_{Y=0}$. Sampling from the red part of distributions, counterfactual images do not contain gray hair. Sampling from the blue part of distributions, counterfactual images have gray hair.
• Figure 4: The causal diagram of the in $\mathcal{M}^*$ in \ref{['ex:face-ascm']}
• Figure 5: The ANCM network structure for \ref{['ex:face-ascm']}.

Theorems & Definitions (25)

• Example 1.1
• Definition 1.2: Structure Causal Model(SCM)
• Definition 1.3: Optimal Counterfactual Bounds
• Definition 1.4: $\mathcal{G}$-Constrained Neural Causal Model ($\mathcal{G}$-NCM)
• Definition 2.1: Augmented Structure Causal Model
• Example 2.2
• Corollary 3.0: Image Causal Hierarchy Theorem
• Example 3.1
• Definition 3.2: Identifiability
• Theorem 3.3: ID