Alterbute: Editing Intrinsic Attributes of Objects in Images

TL;DR

Alterbute presents a diffusion-based framework for editing intrinsic object attributes (color, texture, material, shape) while preserving perceived identity and scene context. It introduces a relaxed supervised training objective that learns from edits involving both intrinsic and extrinsic attributes and uses Visual Named Entities (VNEs) for identity conditioning via an automated vision-language pipeline. The model is trained on a 1×2 grid with an identity reference, an attribute prompt, and background+mask context, enabling inference-time edits that keep extrinsic factors fixed. Extensive experiments and user studies show strong identity preservation and accurate intrinsic edits, outperforming both general-purpose and attribute-specific baselines and demonstrating scalable, single-model intrinsic editing across diverse object categories.

Abstract

We introduce Alterbute, a diffusion-based method for editing an object's intrinsic attributes in an image. We allow changing color, texture, material, and even the shape of an object, while preserving its perceived identity and scene context. Existing approaches either rely on unsupervised priors that often fail to preserve identity or use overly restrictive supervision that prevents meaningful intrinsic variations. Our method relies on: (i) a relaxed training objective that allows the model to change both intrinsic and extrinsic attributes conditioned on an identity reference image, a textual prompt describing the target intrinsic attributes, and a background image and object mask defining the extrinsic context. At inference, we restrict extrinsic changes by reusing the original background and object mask, thereby ensuring that only the desired intrinsic attributes are altered; (ii) Visual Named Entities (VNEs) - fine-grained visual identity categories (e.g., ''Porsche 911 Carrera'') that group objects sharing identity-defining features while allowing variation in intrinsic attributes. We use a vision-language model to automatically extract VNE labels and intrinsic attribute descriptions from a large public image dataset, enabling scalable, identity-preserving supervision. Alterbute outperforms existing methods on identity-preserving object intrinsic attribute editing.

Figures (27)

• Figure 1: Overview of Alterbute. Alterbute fine-tunes a diffusion model for text-guided intrinsic attribute editing. Left (Training): Inputs are arranged in a $1 \times 2$ image grid. The left half contains the noisy latent of the target image, while the right half contains a reference image sampled from the same VNE cluster. The model is conditioned on this reference image, a textual prompt describing the desired intrinsic attributes, a background image, and a binary object mask (both represented as grids). The diffusion loss is applied only to the left half to focus the learning on the edited region. Right (Inference): Using the same architecture (grid omitted for clarity), Alterbute edits the input image directly by reusing its original background and mask. For color, texture, or material edits, we use precise segmentation masks (top). For shape edits where the target geometry is unknown, we use coarse bounding-box masks (bottom).
• Figure 2: We use Gemini to assign textual VNE labels to objects detected in OpenImages. VNE objects (e.g., “Porsche 911 Carrera”) are grouped into VNE clusters, while unlabeled instances are filtered out. Example VNE clusters are shown on the right. For each VNE-labeled object, we additionally prompt Gemini to extract intrinsic attribute descriptions, which serve as textual prompts $p$ during training.
• Figure 3: Qualitative results across intrinsic editing tasks.Alterbute successfully edits a variety of intrinsic attributes.
• Figure 4: Qualitative comparison. Baselines often fail to apply the desired edit or preserve identity. In contrast, Alterbute produces edits that faithfully reflect the target attribute while maintaining object identity.
• Figure 5: Comparison with attribute-specific editors. On the left, for MimicBrush and MaterialFusion, we show the input image, reference image, and their edited output. On the right, we present the result produced by Alterbute.