Consolidating Attention Features for Multi-view Image Editing

TL;DR

This paper tackles 3D inconsistencies that arise when editing multi-view image sets with diffusion models. It introduces QNeRF, a NeRF trained on self-attention query features from the diffusion process, to render 3D-consistent query guidance that is softly injected back into the network. The approach uses an interval-based, progressive consolidation strategy to align geometry across views while preserving appearance, yielding higher fidelity edits with fewer artifacts. Practically, this enables reliable, geometry-aware editing of articulated objects and scenes with fewer visual inconsistencies across viewpoints.

Abstract

Large-scale text-to-image models enable a wide range of image editing techniques, using text prompts or even spatial controls. However, applying these editing methods to multi-view images depicting a single scene leads to 3D-inconsistent results. In this work, we focus on spatial control-based geometric manipulations and introduce a method to consolidate the editing process across various views. We build on two insights: (1) maintaining consistent features throughout the generative process helps attain consistency in multi-view editing, and (2) the queries in self-attention layers significantly influence the image structure. Hence, we propose to improve the geometric consistency of the edited images by enforcing the consistency of the queries. To do so, we introduce QNeRF, a neural radiance field trained on the internal query features of the edited images. Once trained, QNeRF can render 3D-consistent queries, which are then softly injected back into the self-attention layers during generation, greatly improving multi-view consistency. We refine the process through a progressive, iterative method that better consolidates queries across the diffusion timesteps. We compare our method to a range of existing techniques and demonstrate that it can achieve better multi-view consistency and higher fidelity to the input scene. These advantages allow us to train NeRFs with fewer visual artifacts, that are better aligned with the target geometry.

Figures (14)

• Figure 1: Given an object-centric multi-view image set (center), we edit all images simultaneously (left and right), using 3D geometric control, such as changing the body skeleton. To promote consistency across different views, we leverage an image diffusion model and introduce QNeRF, a query feature space neural radiance field, to progressively consolidate attention features during the generation process.
• Figure 2: Editing multi-view images of a boot, with a loose depth map bhat2023loosecontrol. We show a sample of three images from the set.
• Figure 3: The first and third rows show images captured from different viewpoints. When these are individually edited using ControlNet zhang2023adding and MasaCtrl cao2023masactrl, inconsistencies arise. Note the shape of the lamp (top) or the distance of the foot from the wall (bottom). Images were edited using 2D controls projected from a shared 3D model (skeleton, box). The leftmost column shows controls corresponding to view 1.
• Figure 4: We simultaneously generate multi-view edited images with a diffusion model. To consolidate the images, along the denoising process we (1) extract self-attention queries from the network, (2) train a NeRF (termed QNeRF) on the extracted queries and render consolidated queries, and (3) softly inject the rendered queries back to the network for each view. We repeat these steps throughout the denoising process.
• Figure 5: The architecture of QNeRF. Nine heads are attached to the base network, to produce queries corresponding to nine self-attention layers of the diffusion model. Each group of heads corresponds to a self-attention layer of a certain resolution, and the number displayed above the arrow represents the number of channels in that group (1280, 640, 320).