OneTo3D: One Image to Re-editable Dynamic 3D Model and Video Generation

TL;DR

OneTo3D tackles the challenge of producing re-editable dynamic 3D content and long semantic 3D videos from a single image. It fuses Gaussian Splatting-based implicit 3D reconstruction with explicit armature-based editing, using Zero-1-to-3 as a lightweight base and Blender for animation, augmented by an automatic self-adaptive armature binding and a text-to-action interpreter. The approach translates user instructions into detailed motion commands and keyframes, enabling precise pose control while maintaining high rendering speed and lower VRAM requirements compared to pure diffusion or fully implicit methods. This hybrid pipeline aims to deliver practical, near real-time, re-editable 3D video generation from minimal input, with open-source code to encourage adoption and extension.

Abstract

One image to editable dynamic 3D model and video generation is novel direction and change in the research area of single image to 3D representation or 3D reconstruction of image. Gaussian Splatting has demonstrated its advantages in implicit 3D reconstruction, compared with the original Neural Radiance Fields. As the rapid development of technologies and principles, people tried to used the Stable Diffusion models to generate targeted models with text instructions. However, using the normal implicit machine learning methods is hard to gain the precise motions and actions control, further more, it is difficult to generate a long content and semantic continuous 3D video. To address this issue, we propose the OneTo3D, a method and theory to used one single image to generate the editable 3D model and generate the targeted semantic continuous time-unlimited 3D video. We used a normal basic Gaussian Splatting model to generate the 3D model from a single image, which requires less volume of video memory and computer calculation ability. Subsequently, we designed an automatic generation and self-adaptive binding mechanism for the object armature. Combined with the re-editable motions and actions analyzing and controlling algorithm we proposed, we can achieve a better performance than the SOTA projects in the area of building the 3D model precise motions and actions control, and generating a stable semantic continuous time-unlimited 3D video with the input text instructions. Here we will analyze the detailed implementation methods and theories analyses. Relative comparisons and conclusions will be presented. The project code is open source.

Figures (13)

• Figure 1: Architecture of OneTo3D.
• Figure 2: Architecture of OneTo3D.
• Figure 3: Removing the background with color groups.
• Figure 4: Removing the background with edge detection and armature structure or body key-points detection and trunks domain segmentation. The background is usually colorful and complex in RGB other formats. Using a block with one single color to present a colorful pixel. Black bold simple human-like outline present the designed novel armature of the object, which including the key points detection data. Sub-figure (c) represents the trunks domain that contains the main components of the object trunk.
• Figure 5: Removing the background with edges detection based on convolution computation gradient changing calculation.