LouvreSAE: Sparse Autoencoders for Interpretable and Controllable Style Transfer

TL;DR

This work defines an operational notion of artistic style and introduces LouvreSAE, an art-specific Sparse Autoencoder trained on CLIP embeddings to extract a sparse, interpretable dictionary of style concepts. Style profiles are constructed from a handful of reference artworks and steer generative models in latent space without fine-tuning, enabling lightweight, interpretable, and composable style transfer. Evaluations on ArtBench10 show LouvreSAE achieves strong style fidelity while delivering 1.7–20x speedups over baselines, and qualitative results demonstrate clear, human-understandable control over stylistic elements. The approach emphasizes interpretability through a taxonomy of concepts and an autointerpretability pipeline, offering a practical path toward fine-grained, user-controlled style manipulation in generative systems.

Abstract

Artistic style transfer in generative models remains a significant challenge, as existing methods often introduce style only via model fine-tuning, additional adapters, or prompt engineering, all of which can be computationally expensive and may still entangle style with subject matter. In this paper, we introduce a training- and inference-light, interpretable method for representing and transferring artistic style. Our approach leverages an art-specific Sparse Autoencoder (SAE) on top of latent embeddings of generative image models. Trained on artistic data, our SAE learns an emergent, largely disentangled set of stylistic and compositional concepts, corresponding to style-related elements pertaining brushwork, texture, and color palette, as well as semantic and structural concepts. We call it LouvreSAE and use it to construct style profiles: compact, decomposable steering vectors that enable style transfer without any model updates or optimization. Unlike prior concept-based style transfer methods, our method requires no fine-tuning, no LoRA training, and no additional inference passes, enabling direct steering of artistic styles from only a few reference images. We validate our method on ArtBench10, achieving or surpassing existing methods on style evaluations (VGG Style Loss and CLIP Score Style) while being 1.7-20x faster and, critically, interpretable.

Figures (11)

• Figure 1: Overview of LouvreSAE. Our method performs style transfer using an art-specific sparse autoencoder (SAE) on top of latent embeddings of a pre-trained image-to-image (I2I) models, as shown on the left. The SAE enables us to construct style profiles, which contain interpretable concepts and intensities that can be inspected and adjusted. Shown on the right are qualitative, representative examples of images generated with our approach, where a photograph (content image) was turned into styles of six prominent artists.
• Figure 2: LouvreSAE Training & Inference Stages. The method comprises three stages: (a) Backbone SAE Training, where the SAE is presented with a wide range of data sources (including both generic and art-specific datasets); (b) Style Profile Encoding, where a handful of sparse codes obtained from samples of the target style are overlapped to find overlapping concepts that make up the style; and (c) Generation Steering, where the style profile us used as a steering vector during image-to-image pipeline inference.
• Figure 3: Examples of SAE Concepts. Shown above are the autointerpretability name and label, the prototype visualization, and four representative exemplars for two representative concepts learned by LouvreSAE-20.
• Figure 4: Qualitative Comparison of SAE Baselines. We compare LouvreSAE-20K with the FeatureLab SAE by swapping the backbone SAE module of the Kandinsky 2.2 pipeline with LouvreSAE Steering. In each row, the content image is transferred to the style of a different artist: (from top to bottom) Alfred Altdorfor, Durer, Matisse, and Hiroshige.
• Figure 5: Style Profile Comparison. Paul Cézanne and Pierre-Auguste Renoir, two Impressionist artists with visually similar styles, nevertheless exhibit distinct and interpretable style profiles. Shown here is a subset of concepts they share, with the difference in concept intensity visualized. Additional unique concepts and further differences also exist beyond those displayed.