IntrinsicEdit: Precise generative image manipulation in intrinsic space

TL;DR

IntrinsicEdit introduces a unified intrinsic-space editing workflow that operates on RGB$ ightarrow$X intrinsic decompositions and an X$ ightarrow$RGB neural renderer. By performing exact diffusion inversion and a novel prompt-tuning plus channel-transfer strategy, it preserves image identity while enabling precise, localized edits across materials, object insertion/removal, and relighting without additional data or model fine-tuning. The approach addresses identity drift and inter-channel entanglement, delivering state-of-the-art results on indoor scenes with automatic illumination adjustments. This framework advances practical, pixel-precise editing using diffusion models, with implications for content creation and computational photography, while noting limitations in outdoor scenes, metals, and complex light transport, and highlighting ethical considerations for misuse of realistic manipulation.

Abstract

Generative diffusion models have advanced image editing with high-quality results and intuitive interfaces such as prompts and semantic drawing. However, these interfaces lack precise control, and the associated methods typically specialize on a single editing task. We introduce a versatile, generative workflow that operates in an intrinsic-image latent space, enabling semantic, local manipulation with pixel precision for a range of editing operations. Building atop the RGB-X diffusion framework, we address key challenges of identity preservation and intrinsic-channel entanglement. By incorporating exact diffusion inversion and disentangled channel manipulation, we enable precise, efficient editing with automatic resolution of global illumination effects -- all without additional data collection or model fine-tuning. We demonstrate state-of-the-art performance across a variety of tasks on complex images, including color and texture adjustments, object insertion and removal, global relighting, and their combinations.

Figures (13)

• Figure 1: RGB$\leftrightarrow$X overview. An RGB$\rightarrow$X diffusion model decomposes a given image into intrinsic channels, while a complementary neural rendering X$\rightarrow$RGB diffusion model composes channels into an image zeng2024rgb. The complete image-to-image RGB$\rightarrow$X$\rightarrow$RGB pipeline promises semantic editing with pixel precision by manipulating the channels before recomposition. Unfortunately, the models' generative nature causes random identity shifts in the resynthesized image, and successful editing requires adjusting multiple entangled channels, hindering usability. We address both these issues to unlock the image-editing potential of RGB$\leftrightarrow$X.
• Figure 2: IntrinsicEdit overview. We outline our intrinsic-space editing pipeline, here showing the removal of flowers by manipulating the albedo channel. ➀ We run RGB$\rightarrow$X to decompose the input image into intrinsic channels. ➁ We tune the prompt embedding to the image and channels (\ref{['sec:prompt_tuning']}). We also select a subset of channels for editing (here albedo only); any other channels that are entangled with that subset for the desired edit are transferred to the prompt and subsequently dropped (\ref{['sec:channel_transfer']}). This step allows us to edit a single channel while preserving information from the rest. ➂ We perform exact X$\rightarrow$RGB inversion w.r.t. the remaining conditions, i.e. kept channels and optimized prompt. This step finds the noise map that, together with the conditions, accurately reconstructs the input image (\ref{['sec:inversion']}). ➃ We can now perform the desired edit by manipulating the selected channels. ➄ Finally, we feed those channels to the X$\rightarrow$RGB model, along with the optimized prompt and inverted noise, to synthesize the edited image (\ref{['sec:image_editing']}). This pipeline allows us to alter only certain image modalities (e.g. material), while automatically propagating the changes to a realistic result and preserving untouched aspects.
• Figure 3: Material editing. We compare our method against two intrinsic-image methods: original RGB$\rightarrow$X$\rightarrow$RGBzeng2024rgb and intrinsic image diffusion kocsis2024intrinsic, and two prompt-based methods: Grounded-Instruct-Pix2Pix Groundedip2p and TurboEdit deutch2024turboedit. Prompt-based methods fail to provide fine-grained control. Ours is the only one that allows for precise manipulation of individual material properties, preserving identity and harmonizing the edits much better than prior intrinsic-space approaches. Notice the red wall in the top row matching the original material properties while correctly adjusting the color, including the reflection on the counter. The second row shows texture editing on the armchair and addition of two pillows, preserving the lighting and scene identity. In the third row, our approach automatically extends the wooden floor and preserves the kitchen island color despite editing only the normal map. The bottom row shows roughness editing, making the floor more matte and adjusting the reflections.
• Figure 4: Relighting. In the top two rows we generate a new irradiance channel via prompting as described in \ref{['sec:relighting']}. In the bottom row we generate novel irradiance maps using the volumetric shading model of OutCast griffiths2022outcast. Our method handles the new lighting condition more naturally than original RGB$\rightarrow$X$\rightarrow$RGB relighting zeng2024rgb, even when the change is drastic (second row), and better preserves the identity of the scene content.
• Figure 5: Object removal. We compare against original RGB$\rightarrow$X$\rightarrow$RGBzeng2024rgb, Photoshop generative fill photoshop, and Stable Diffusion XL inpainting sdxl-inpainting. Without being specialized for this task, our method performs on par with or better than prior work. In the top row, notice the correct removal of the plant reflection from the marble floor. In the second and fourth rows, the table's texture is preserved better. In the third row, our method successfully removes the left cup, including the shadow it casts on the other cup, while previous methods even struggle to remove the cup completely.