Style Feature Extraction Using Contrastive Conditioned Variational Autoencoders with Mutual Information Constraints

TL;DR

The paper addresses the challenge of extracting fine-grained style features from unlabeled data by decomposing representations into content ($z_{\text{content}}$) and style ($z_{\text{style}}$) using a CVAE conditioned on CL-derived content and reinforced by a mutual-information constraint. It trains a CL backbone to provide style-insensitive content features and then learns a CVAE to capture style, with a MINE-based MI estimator encouraging independence between $z_{\text{content}}$ and $z_{\text{style}}$ via a JS-based lower bound. The approach is validated on MNIST and Google Fonts, demonstrating isolation of style factors and controllable style generation, and extended to Imagenette and DAISO-100 to show scalability to real-world data. The work offers a practical framework for unsupervised style analysis and transfer across diverse visual domains, with potential improvements through GAN integration and extension to other data modalities.

Abstract

Extracting fine-grained features such as styles from unlabeled data is crucial for data analysis. Unsupervised methods such as variational autoencoders (VAEs) can extract styles that are usually mixed with other features. Conditional VAEs (CVAEs) can isolate styles using class labels; however, there are no established methods to extract only styles using unlabeled data. In this paper, we propose a CVAE-based method that extracts style features using only unlabeled data. The proposed model consists of a contrastive learning (CL) part that extracts style-independent features and a CVAE part that extracts style features. The CL model learns representations independent of data augmentation, which can be viewed as a perturbation in styles, in a self-supervised manner. Considering the style-independent features from the pretrained CL model as a condition, the CVAE learns to extract only styles. Additionally, we introduce a constraint based on mutual information between the CL and VAE features to prevent the CVAE from ignoring the condition. Experiments conducted using two simple datasets, MNIST and an original dataset based on Google Fonts, demonstrate that the proposed method can efficiently extract style features. Further experiments using real-world natural image datasets were also conducted to illustrate the method's extendability.

Figures (36)

• Figure 1: Overview of the proposed method. The VAE conditioned by the CL model extracts the style features. In the training procedure, the estimated mutual information of the two feature vectors $z_\text{content}$ and $z_\text{style}$ is evaluated to encourage the features to be statistically independent.
• Figure 2: Example images from the Google Fonts dataset.
• Figure 3: MNIST
• Figure 4: Google Fonts
• Figure 6: MNIST