HawkI: Homography & Mutual Information Guidance for 3D-free Single Image to Aerial View

TL;DR

HawkI addresses the problem of generating realistic aerial-view images from a single image and text without 3D or multi-view training data. It combines test-time optimization of text embeddings and diffusion model LoRA with an IPM-based homography to introduce aerial cues, and introduces mutual information guidance to maintain semantic fidelity to the input despite viewpoint changes. The approach achieves a favorable bias-variance trade-off, outperforming text+exemplar baselines and rivaling 3D-based NVS methods on synthetic and real data while being 3D-free at training and inference. This results in practical benefits for synthetic data generation and downstream UAV tasks, with potential for further gains by integrating 3D priors and expanding controllable viewpoints.

Abstract

We present HawkI, for synthesizing aerial-view images from text and an exemplar image, without any additional multi-view or 3D information for finetuning or at inference. HawkI uses techniques from classical computer vision and information theory. It seamlessly blends the visual features from the input image within a pretrained text-to-2Dimage stable diffusion model with a test-time optimization process for a careful bias-variance trade-off, which uses an Inverse Perspective Mapping (IPM) homography transformation to provide subtle cues for aerialview synthesis. At inference, HawkI employs a unique mutual information guidance formulation to steer the generated image towards faithfully replicating the semantic details of the input-image, while maintaining a realistic aerial perspective. Mutual information guidance maximizes the semantic consistency between the generated image and the input image, without enforcing pixel-level correspondence between vastly different viewpoints. Through extensive qualitative and quantitative comparisons against text + exemplar-image based methods and 3D/ multi-view based novel-view synthesis methods on proposed synthetic and real datasets, we demonstrate that our method achieves a significantly better bias-variance trade-off towards generating high fidelity aerial-view images.Code and data is available at https://github.com/divyakraman/HawkI2024.

Figures (21)

• Figure 1: HawkI generates aerial-view images from a text description and an exemplar input image. It builds on a text to 2D image stable diffusion model and does not require any additional 3D or multi-view information at fine-tuning or inference.
• Figure 2: Overview. HawkI generates aerial-view images, using a text description and a single image $I_{S}$ as supervisory signals. It builds on a pretrained text-to-image diffusion model, and does not use any 3D or multi-view information. It performs test-time finetuning to optimize the text embedding and the diffusion model to reconstruct the input image and its inverse perspective mapping in close vicinity. Such a mechanism enables the incorporation of image specific knowledge within the model, while retaining its imaginative capabilities (or variance). At inference, HawkI uses mutual information guidance to maximize the information between the probability distributions of the generated image and $I_{S}$, to generate a high-fidelity aerial-view image.
• Figure 3: Compared to state-of-the-art text + exemplar image based methods, HawkI is able to generate images that are "more aerial", while being consistent with the input image. The top three images are from the HawkI-Syn dataset, the bottom three images are from the HawkI-Real dataset.
• Figure 4: HawkI achieves the best viewpoint-fidelity trade-off amongst prior work on text + exemplar image based aerial-view synthesis, on various quantitative metrics indicate of text-alignment (for viewpoint and a broad description of the scene) and image alignment (for fidelity w.r.t. input image).
• Figure 5: (Left figure.) Ablation experiments show that Inverse Perspective Mapping helps in the generation of images that are aerial, mutual information guidance helps in preserving the contents w.r.t. input image. (Right figure.) We compare with latest related work on novel view synthesis: Zero-1-to-3liu2023zero and Zero123++ shi2023zero123++. Both methods use the pretrained text-to-2D-image stable diffusion model along with the 800k 3D objects dataset, Objaverse deitke2023objaverse, for training. Our method uses just the pretrained text-to-2D-image stable diffusion model to generate better results for the task of aerial view synthesis, guided by text and a single input image.