PreciseCam: Precise Camera Control for Text-to-Image Generation

TL;DR

PreciseCam addresses the lack of precise camera control in text-to-image diffusion by introducing a PF-US per-pixel view representation conditioned on four intuitive parameters $\Omega=(\text{roll},\text{pitch},\text{vFoV},\xi)$. It leverages a ControlNet-adapted diffusion backbone to map these parameters to per-pixel information, enabling accurate view changes without requiring 3D scenes or multi-view data. A new dataset of 57,380 RGB images with ground-truth PF-US parameters is built from 360° panoramas to train the conditioning module. Results show robust, accurate camera control across extrinsic and intrinsic parameters, with competitive prompt fidelity and practical applications in background generation, video conditioning, and extended ControlNet composition. The work provides a publicly available dataset, model, and code, advancing precise camera-aware generation for artistic and realistic imagery alike.

Abstract

Images as an artistic medium often rely on specific camera angles and lens distortions to convey ideas or emotions; however, such precise control is missing in current text-to-image models. We propose an efficient and general solution that allows precise control over the camera when generating both photographic and artistic images. Unlike prior methods that rely on predefined shots, we rely solely on four simple extrinsic and intrinsic camera parameters, removing the need for pre-existing geometry, reference 3D objects, and multi-view data. We also present a novel dataset with more than 57,000 images, along with their text prompts and ground-truth camera parameters. Our evaluation shows precise camera control in text-to-image generation, surpassing traditional prompt engineering approaches. Our data, model, and code are publicly available at https://graphics.unizar.es/projects/PreciseCam2024.

Figures (16)

• Figure 1: Our approach enhances the artistic expression of text-to-image generative models by incorporating precise control over camera angles and lens distortion effects. Left: Our input consists of a standard text prompt along with extrinsic (roll and pitch) and intrinsic (vertical field of view and distortion $\xi$) camera parameters, which are translated into a suitable and efficient representation for learning camera views. Right: Examples varying roll (top) and pitch (bottom) with the same prompt, while keeping the remaining camera parameters fixed.
• Figure 2: Red boxes: Current approaches rely on trial-and-error prompt engineering or generalistic tags, offering limited camera control in text-to-image generative AI. Others use 3D representations of the scenes, from which depth or edge maps are obtained, but this imposes strict constraints on the resulting image, limiting flexibility. Green box: Our method relies on two extrinsic and two intrinsic camera parameters, user-provided; we then obtain the effects that such camera parameters impose on the appearance of each pixel in the final image, encoded in a 2D PF-US map (see text for details). This allows for precise, fine-tuned camera control, thus enhancing creativity.
• Figure 3: Left and center: PF-US camera view representation. PF-US camera parameters (roll, pitch, vertical FoV, and distortion $\xi$) and associated maps corresponding to two example images. PF-US maps encode, for each pixel, a latitude value $\mathbf{\varphi_x}$ (blue-red color map with contour lines) and a unit up-vector $\mathbf{u_x}$ (green arrows). Right: Dataset generation. We generate training data in the form of images with ground-truth PF-US maps by leveraging 360$^\circ$ images. We sample our camera parameters and obtain the corresponding cropped region for each sampled quartet (as examples, the highlighted regions in red and yellow yield the two examples shown on the left).
• Figure 4: Training of our proposed approach for precise camera control. Text-to-image generation with a UNet-based diffusion model (SDXL in our implementation) is conditioned with a PF-US map representing the desired camera parameters; conditioning is learned by means of a ControlNet-based module. During inference, only the middle block output of this module is injected into SDXL (shown in red, see text for more details).
• Figure 5: Extrinsic Parameter Control. Images generated by varying the extrinsic camera parameters while keeping the rest of the conditioning fixed, showing consistent adherence to the camera specification. Top row: Variation in pitch, effectively shifting the view from looking downward to upward; fixed parameters are $(\textrm{roll}, \textrm{vFoV}, \xi)$ = (0º, 80º, 0.1). Right: Variation in roll, tilting the view from left to right; $(\textrm{pitch}, \textrm{vFoV}, \xi)$ = (0º, 80º, 0.1). Insets show the corresponding PF-US maps, and text prompts are shown in italicized text.