3D Object Manipulation in a Single Image using Generative Models

TL;DR

OMG3D introduces a unified diffusion-based framework that converts 2D objects into editable 3D proxies to enable both static edits and dynamic motion. It combines CustomRefiner, a per-concept diffusion-based texture refinement with differentiable rasterization, and IllumiCombiner, a lighting processing module that estimates and corrects background lighting to produce realistic shadows. The method enables end-to-end rendering of photorealistic 3D manipulated objects from a single image and demonstrates superior performance over state-of-the-art baselines in both image editing and video generation, validated by GPT-4o and user studies on a single RTX 3090. These contributions advance realistic 3D-aware editing and animation from 2D inputs with practical hardware requirements and broad applicability in design, AR/VR, and film. Future work targets more complex scenes, dynamic backgrounds, and interactions between multiple objects.

Abstract

Object manipulation in images aims to not only edit the object's presentation but also gift objects with motion. Previous methods encountered challenges in concurrently handling static editing and dynamic generation, while also struggling to achieve fidelity in object appearance and scene lighting. In this work, we introduce \textbf{OMG3D}, a novel framework that integrates the precise geometric control with the generative power of diffusion models, thus achieving significant enhancements in visual performance. Our framework first converts 2D objects into 3D, enabling user-directed modifications and lifelike motions at the geometric level. To address texture realism, we propose CustomRefiner, a texture refinement module that pre-train a customized diffusion model, aligning the details and style of coarse renderings of 3D rough model with the original image, further refine the texture. Additionally, we introduce IllumiCombiner, a lighting processing module that estimates and corrects background lighting to match human visual perception, resulting in more realistic shadow effects. Extensive experiments demonstrate the outstanding visual performance of our approach in both static and dynamic scenarios. Remarkably, all these steps can be done using one NVIDIA 3090. Project page is at https://whalesong-zrs.github.io/OMG3D-projectpage/

Figures (10)

• Figure 1: Applications of OMG3D: combine different concepts with background images.
• Figure 2: Overview of our method. The teaser image provides a visual summary of the key contribution of this work.
• Figure 3: We demonstrates the workflow of OMG3D (§ Section \ref{['sec:omg3d']}). OMG3D has two key modules: (1) CustomRefiner (§ Section \ref{['sec:texture']}), and (2) IllumiCombiner (§ Section \ref{['sec:light']}). The former leverages a customized diffusion model to optimize the rendering viewpoints and subsequently improve the model's textures. The latter uses color information from the original image and predefined light sources to correct the color and enhance the intensity of the estimated light, ensuring natural lighting in the final image.
• Figure 4: Qualitative comparison with image editing methods (§ Section \ref{['sec:evaluation']}). Text-based methods fail to achieve the target action and maintain appearance, while Image Sculpting faces partial object loss.
• Figure 5: Qualitative comparison with image animation methods (§ Section \ref{['sec:evaluation']}). * indicates failure cases where the object remains stationary in the video. Distorions are highlighed by red rectangles. The pumpkin's positions in Image Sculpting are changed across frames.