Neural USD: An object-centric framework for iterative editing and control

TL;DR

Neural USD addresses the challenge of precise, iterative object editing in complex scenes by introducing an object-centric conditioning framework inspired by the Universal Scene Descriptor. It represents scenes as per-object assets with appearance, geometry, and pose signals, and uses paired video frames to train encoders and fine-tune pre-trained image models so these signals become disentangled. The approach enables fine-grained control over multiple objects and supports iterative workflows where edits to one object do not undesirably alter others, achieving superior reconstruction and controllability compared to several baselines. By leveraging cross-attention or FiLM conditioning and a flexible asset encoding scheme, Neural USD offers a portable standard that can be integrated with diffusion and transformer-based generators, with strong implications for interactive, multi-object image editing and 3D-aware synthesis.

Abstract

Amazing progress has been made in controllable generative modeling, especially over the last few years. However, some challenges remain. One of them is precise and iterative object editing. In many of the current methods, trying to edit the generated image (for example, changing the color of a particular object in the scene or changing the background while keeping other elements unchanged) by changing the conditioning signals often leads to unintended global changes in the scene. In this work, we take the first steps to address the above challenges. Taking inspiration from the Universal Scene Descriptor (USD) standard developed in the computer graphics community, we introduce the "Neural Universal Scene Descriptor" or Neural USD. In this framework, we represent scenes and objects in a structured, hierarchical manner. This accommodates diverse signals, minimizes model-specific constraints, and enables per-object control over appearance, geometry, and pose. We further apply a fine-tuning approach which ensures that the above control signals are disentangled from one another. We evaluate several design considerations for our framework, demonstrating how Neural USD enables iterative and incremental workflows. More information at: https://escontrela.me/neural_usd .

Figures (16)

• Figure 1: Demo of iterative editing using the finetuned Neural USD model. Given the original image (a), we specify the desired target pose as a 3D bounding box (b). The model is able to precisely 'move' the object to the desired pose while rest of the scene elements remain fairly consistent. Next in (c), we change the appearance (in this case color) while also being in the new pose. Our model is able to handle these multiple requests. In (d), we retain the desired pose from (b) and condition on another office chair's geometry (or depth) and appearance. Our model is able to perform these edits while leaving rest of the scene elements (notably the background) consistent with the original image. Note that using these conditions we can replace an object in the image with another object in any desired pose. In (e), we edit the pose, geometry and background all at the same time. Our model is able to handle these requests simultaneously and generates an image that respects all the given conditions. More details on the conditioning signals and additional examples are in Section \ref{['section:iterative_editing']} and appendix \ref{['sec:iterative_workflows']}
• Figure 2: Neural USD enables computer graphics-style control of image models. A Neural USD represents an image as assets with appearance, geometry, and pose. Fine-tuning adapts pre-trained models to these signals while keeping appearance and geometry pose-invariant.
• Figure 3: Neural USD Overview. a) A Neural USD consists of assets with multiple modalities: appearance, geometry, and pose. b) Pre-trained image models fine-tune on Neural USD, encoding appearance and geometry from a source image and pose from a target image to reconstruct the target. c) At inference, objects' poses, geometry, and appearance can be modified, including the background.
• Figure 4: Neural USD allows users to perform a variety of pose, appearance, and geometry modifications to both the foreground and the background objects.
• Figure 5: Object replacement examples with appearance and geometry conditioning (top) and geometry conditioning (bottom).