Recolour What Matters: Region-Aware Colour Editing via Token-Level Diffusion

Abstract

Colour is one of the most perceptually salient yet least controllable attributes in image generation. Although recent diffusion models can modify object colours from user instructions, their results often deviate from the intended hue, especially for fine-grained and local edits. Early text-driven methods rely on discrete language descriptions that cannot accurately represent continuous chromatic variations. To overcome this limitation, we propose ColourCrafter, a unified diffusion framework that transforms colour editing from global tone transfer into a structured, region-aware generation process. Unlike traditional colour driven methods, ColourCrafter performs token-level fusion of RGB colour tokens and image tokens in latent space, selectively propagating colour information to semantically relevant regions while preserving structural fidelity. A perceptual Lab-space Loss further enhances pixel-level precision by decoupling luminance and chrominance and constraining edits within masked areas. Additionally, we build ColourfulSet, a largescale dataset of high-quality image pairs with continuous and diverse colour variations. Extensive experiments demonstrate that ColourCrafter achieves state-of-the-art colour accuracy, controllability and perceptual fidelity in fine-grained colour editing. Our project is available at https://yangyuqi317.github.io/ColourCrafter.github.io/.

Figures (12)

• Figure 1: Editing results of ColourCrafter under varying reference colours. Each row shows the input image and its edited outputs conditioned on different RGB references. As the reference colours vary smoothly from left to right, ColourCrafter produces continuous and precise recolouring with consistent structure and texture.
• Figure 2: Comparison of different colour editing methods. (a) shows the results of natural language description (first row) and text embedding interpolation (second row); (b) displays the image-driven results using IP-Adapter. In comparison, our proposed method achieves both precise colour control and high structural fidelity.
• Figure 3: Overview of the ColourCrafter pipeline. (1) Dataset construction: Using Flux.1-Kontext, we generate diverse image–colour pairs and employ a Vision–Language Model (VLM) to filter samples for consistency, fidelity, and realism. The corresponding RGB references are extracted to build the high-quality dataset ColourfulSet. (2) Training: The original image, target colour reference, and text prompt are jointly fed into the diffusion model, which is optimised with both Diffusion and Lab-space losses to enhance chromatic accuracy and perceptual consistency. (3) Inference: Given an input image, a RGB reference, and a prompt, ColourCrafter performs fine-grained, structure-preserving, and perceptually natural colour editing.
• Figure 4: Examples from the ColourfulSet dataset. (a) shows edited images from different categories under various target colour references. (b–c) Visualise the colour distribution of ColourfulSet from two perspectives in RGB space, demonstrating its diversity and uniform coverage.
• Figure 5: Comparison with other methods. The first column shows the reference colours and original images. The comparison demonstrates that our method achieves more precise and fine-grained colour editing while preserving structural integrity and background consistency.