Beyond Static Scenes: Camera-controllable Background Generation for Human Motion

TL;DR

This work tackles unbounded image composition by predicting camera view changes and image composition beyond the original frame. It introduces UNIC, a DETR-based framework augmented with a Feature Extrapolation Module (FEM) that predicts latent representations outside the initial view to support out-of-bounds reasoning, optimized with a composition loss $\mathcal{L}_{comp}$ and an extrapolation loss $\mathcal{L}_{extra}$. To train and evaluate effectively, the authors construct unbounded variants of existing cropping datasets (GAICD and CPC), enabling metrics such as $Acc_{K/N}$, $IoU$, and $Disp$ to quantify performance. Empirical results demonstrate that UNIC outperforms state-of-the-art cropping methods on unbounded tasks, while maintaining competitive results on traditional cropping benchmarks, highlighting its practical potential for guiding camera view and composition in real-world photography scenarios. The work also identifies limitations and outlines future directions, including incorporating richer scene cues such as depth and multi-step view adjustments to further enhance robustness.

Abstract

In this paper, we investigate the generation of new video backgrounds given a human foreground video, a camera pose, and a reference scene image. This task presents three key challenges. First, the generated background should precisely follow the camera movements corresponding to the human foreground. Second, as the camera shifts in different directions, newly revealed content should appear seamless and natural. Third, objects within the video frame should maintain consistent textures as the camera moves to ensure visual coherence. To address these challenges, we propose DynaScene, a new framework that uses camera poses extracted from the original video as an explicit control to drive background motion. Specifically, we design a multi-task learning paradigm that incorporates auxiliary tasks, namely background outpainting and scene variation, to enhance the realism of the generated backgrounds. Given the scarcity of suitable data, we constructed a large-scale, high-quality dataset tailored for this task, comprising video foregrounds, reference scene images, and corresponding camera poses. This dataset contains 200K video clips, ten times larger than existing real-world human video datasets, providing a significantly richer and more diverse training resource. Project page: https://yaomingshuai.github.io/Beyond-Static-Scenes.github.io/

Figures (5)

• Figure 1: Illustration of our proposed UNIC for unbounded recommendation of camera view and image composition. On the left is the initial view provided by the user. Given the current view, our model can predict camera operations (e.g., zoom out and the movement) and a image composition solution (e.g., $\mathbf{c}_\mathit{pred}$). The prediction can be executed multiple times until convergence.
• Figure 2: Architecture of the proposed UNIC framework. It adopts a cDETR-like encoder-decoder architecture meng2021conditional to predict aesthetic plausible view $\mathbf{c}_\mathit{pred}$ from initial view $\mathbf{I}_\mathit{init}$. To mitigate the difficulty in predicting $\mathbf{c}_\mathit{pred}$ beyond image borders, a feature extrapolation module is deployed to predict the invisible tokens $\mathcal{Z}_\mathit{pad}$ from visible ones $\mathcal{Z}_\mathit{vis}$. The FEM is supervised by tokens extracted from larger view $\mathbf{I}$ with the exponential moving averaged CNN and encoder during training.
• Figure 3: Qualitative comparison with other methods. Our method goes beyond the border of the image to predict a well-composed region with the main objects in reasonable places.
• Figure 4: Visual comparison for multi-step adjustment. Our model predicts better view with increased adjust steps to approach the ground-truth view within the red box.
• Figure 5: Visualization of failure cases.