ACT-R: Adaptive Camera Trajectories for Single View 3D Reconstruction

TL;DR

ACT-R tackles single-view 3D reconstruction by introducing an adaptive camera-trajectory (ACT) strategy that maximizes occlusion revelation and 3D consistency within a fixed view budget. It builds semantic-difference blocks from slice predictions, plans an orbit that prioritizes high-uncertainty regions, then generates a sequence of novel views with a video diffusion model and reconstructs the object with either NeUS or InstantMesh, all without direct 3D supervision. Across 1,030 objects on the GSO dataset, ACT-R consistently improves reconstruction quality and 2D view fidelity compared to static and random baselines, with ablations highlighting the benefits of VGG16-based semantic differences and multi-view planning. The approach is practical and flexible, enabling plug-and-play use with various diffusion-based view generators and 3D backbones, potentially accelerating robust 3D understanding in applications where full 3D supervision is unavailable.

Abstract

We introduce the simple idea of adaptive view planning to multi-view synthesis, aiming to improve both occlusion revelation and 3D consistency for single-view 3D reconstruction. Instead of producing an unordered set of views independently or simultaneously, we generate a sequence of views, leveraging temporal consistency to enhance 3D coherence. More importantly, our view sequence is not determined by a pre-determined and fixed camera setup. Instead, we compute an adaptive camera trajectory (ACT), forming an orbit, which seeks to maximize the visibility of occluded regions of the 3D object to be reconstructed. Once the best orbit is found, we feed it to a video diffusion model to generate novel views around the orbit, which can then be passed to any multi-view 3D reconstruction model to obtain the final result. Our multi-view synthesis pipeline is quite efficient since it involves no run-time training/optimization, only forward inferences by applying pre-trained models for occlusion analysis and multi-view synthesis. Our method predicts camera trajectories that reveal occlusions effectively and produce consistent novel views, significantly improving 3D reconstruction over SOTA alternatives on the unseen GSO dataset. Project Page: https://mingrui-zhao.github.io/ACT-R/

Figures (15)

• Figure 1: ACT-R, for single-view 3D reconstruction, predicts an adaptive camera trajectory (green) to maximize the visibility of occluded object parts over a fixed sequence length (20 views). The trajectory, obtained in under 10s, is then used by a video generator (e.g., SV3D voleti2024sv3d) to produce a sequence of novel views for multi-view 3D reconstruction, here using NeUS wang2021neus. Compared to a generic trajectory (red), at fixed elevation, ACT-R yields much cleaner results with more faithful recovery of occluded regions.
• Figure 2: Pipeline of our single-view reconstruction method, ACT-R, with adaptive camera trajectories (ACT). We first employ Slice3D wang2024slice3d to produce the slice images of the input object, with the slicing direction from the camera to the object center. Then we compute the semantic difference between the input and its slices by comparing their $512 \times 7 \times 7$ feature maps extracted from VGG16 simonyan2014very. Each difference map $d_i \in [0,1]^{7 \times 7}$ is up-scaled and overlaid onto slice images for beter visualization. Next, we identify the regions that have significant semantic differences (Sec. \ref{['sec:semantic_diff']}), and plan the camera trajectories based on them (Sec. \ref{['sec:traj_plan']}). Finally, we condition SV3D voleti2024sv3d on our planned trajectories, yielding a sequence of views, which can be fed into NeUS wang2021neus or InstantMesh (IM) xu2024instantmesh for multi-view 3D reconstruction.
• Figure 3: Initial single-view inputs to advanced methods such as Trellis and Hunyuan3D (HY) can be improved by one of the synthesized multi-view images from ACT-R, to improve reconstruction results, as highlighted in the colored regions.
• Figure 4: Illustration of camera trajectory planning. Left: Transforming the semantic difference maps into 3D blocks, where lighter yellow indicates greater differences. Right: Visualization of different camera orbits. Red: fixed elevation; Green: variable elevations that capture greater visibility. "diff" and "cam" denote difference and camera, respectively.
• Figure 5: Visualization of slice images predicted by Slice3D wang2024slice3d and planned trajectories (shown in green). Purple arrows indicate the world's up direction. The 3D blocks (i.e., semantics difference blocks) roughly represent the input shape, using colors to highlight their semantic differences from the input view. The numbers in brackets show the elevation changes in each segment of the orbit.