Color Transfer with Modulated Flows

TL;DR

This work introduces ModFlows, a color-transfer method that casts color adjustment as an invertible transport in RGB using rectified neural ODEs. By learning a shared intermediate distribution and an encoder that predicts modulated flow weights, the model generalizes to new content–style pairs without retraining and processes 4K images with strong style and content fidelity. The two-stage training—building a rectified-flow dataset and distilling an encoder to predict flow weights—yields an image-palette embedding capable of rapid inference. Quantitative results on diverse datasets show competitive to state-of-the-art style transfer performance with favorable artifact control, and the 4K capability broadens practical applicability. The framework further points to broader uses in distribution-to-distribution transfers beyond color and to applications requiring invertible, data-driven transport in RGB space.

Abstract

In this work, we introduce Modulated Flows (ModFlows), a novel approach for color transfer between images based on rectified flows. The primary goal of the color transfer is to adjust the colors of a target image to match the color distribution of a reference image. Our technique is based on optimal transport and executes color transfer as an invertible transformation within the RGB color space. The ModFlows utilizes the bijective property of flows, enabling us to introduce a common intermediate color distribution and build a dataset of rectified flows. We train an encoder on this dataset to predict the weights of a rectified model for new images. After training on a set of optimal transport plans, our approach can generate plans for new pairs of distributions without additional fine-tuning. We additionally show that the trained encoder provides an image embedding, associated only with its color style. The presented method is capable of processing 4K images and achieves the state-of-the-art performance in terms of content and style similarity. Our source code is available at https://github.com/maria-larchenko/modflows

Figures (7)

• Figure 1: A proposed scheme of training and inference. Color transfer is a composition of a forward content and an inverse style flows applied to the content image.
• Figure 2: Qualitative comparison. Examples from Unsplash Lite test set. Our model achieves the most exact match with the reference palette without visible distortion.
• Figure 3: Colorless content metrics. The choice of the best content metric is not obvious. Edges detection by HED model xie15hed grasps mostly the main objects of a scene, while canny LDC soria2022ldc images are capturing the too detailed edges. Both of them are not sensitive to low-frequency artifacts. To show the absence of such artifacts in the Modflows we additionally compute similarity scores between the normalized grayscale images, which are processed to have a linear intensity histogram through histogram matching, and the depth maps gui2024depthfm.
• Figure 4: Search for similar color styles in the Unsplash Lite dataset (25k images). The top and second rows show search results based on the output of the ModFlows (B6) model. The third and last rows display results based on image statistics, specifically flattened vectors representing the first and second centered moments of the color distribution.
• Figure 5: Ablation study. ModFlows models reach a better trade-off between style and content similarity when compared to dataset models used in their training.