Counterfactual Generation with Identifiability Guarantees

TL;DR

The paper addresses counterfactual generation under domain-varying content-style dependence by proving identifiability of content and style under relative sparsity, without requiring paired data or style labels. It introduces MATTE, a VAE-based framework with flow-based modules that model causal influences and enable style intervention while preserving content across domains. Theoretical guarantees for recovering the content and style subspaces are supported by extensive experiments on four-domain sentiment transfer, where MATTE achieves state-of-the-art requires content preservation and fluent, style-consistent transfers. The work provides a principled, scalable approach to cross-domain counterfactual generation and discusses practical limitations and avenues for extending sparsity-based identifiability to other modalities.

Abstract

Counterfactual generation lies at the core of various machine learning tasks, including image translation and controllable text generation. This generation process usually requires the identification of the disentangled latent representations, such as content and style, that underlie the observed data. However, it becomes more challenging when faced with a scarcity of paired data and labeling information. Existing disentangled methods crucially rely on oversimplified assumptions, such as assuming independent content and style variables, to identify the latent variables, even though such assumptions may not hold for complex data distributions. For instance, food reviews tend to involve words like tasty, whereas movie reviews commonly contain words such as thrilling for the same positive sentiment. This problem is exacerbated when data are sampled from multiple domains since the dependence between content and style may vary significantly over domains. In this work, we tackle the domain-varying dependence between the content and the style variables inherent in the counterfactual generation task. We provide identification guarantees for such latent-variable models by leveraging the relative sparsity of the influences from different latent variables. Our theoretical insights enable the development of a doMain AdapTive counTerfactual gEneration model, called (MATTE). Our theoretically grounded framework achieves state-of-the-art performance in unsupervised style transfer tasks, where neither paired data nor style labels are utilized, across four large-scale datasets. Code is available at https://github.com/hanqi-qi/Matte.git

Figures (6)

• Figure 1: The data generation process: The grey shading indicates the variable is observed. The observed variable (i.e., text) $\mathbf{x}$ is generated from content $\mathbf{c}$ and style $\mathbf{s}$. Both content $\mathbf{c}$ and style $\mathbf{s}$ are influenced by the domain variable $\mathbf{u}$ and the content also influences the style. $\tilde{\mathbf{s}}$ is the exogenous variable of $\mathbf{s}$, representing the independent information of $\mathbf{s}$.
• Figure 2: Sparsity in $\mathbf{J}_{g}$.
• Figure 3: Our VAE-based framework -- MATTE. During training, the input $\mathbf{x}$ is fed to the encoder $f_{\text{enc}}$ to derive the latent variable $\mathbf{z}=[\mathbf{c},\mathbf{s}]$, which is then passed to the decoder $g_{\text{dec}}$ for reconstruction. Flow modules, denoted as $r_{c}$ and $r_{s}$, are implemented to model the causal influences on $\mathbf{c}$ and $\mathbf{s}$ respectively, which yields the creation of exogenous variables $\tilde{\mathbf{c}}$ and $\tilde{\mathbf{s}}$. To generate transferred data $\mathbf{x}_{\text{transfer}}$, we intervene on the style exogenous variable $\tilde{\mathbf{s}}$ while keeping the original content variable $\mathbf{c}$ unchanged (indicated by the green arrows).
• Figure 4: Histograms of negative log-likelihood (NLL) of 1000 Amazon test samples evaluated on the original latent variable and intervened ones. left: flips $\mathbf{s}$, right flips $\tilde{\mathbf{s}}$. The table shows the corresponding sentences.
• Figure 5: The style variables of sentences in past-tense (blue) and present-tense (red) following a UMAP projection. Left: CPVAE; Right: MATTE.

Theorems & Definitions (6)

• Theorem 1
• Theorem 2
• Theorem 2
• proof
• Theorem 2
• proof