Unconstrained Large-scale 3D Reconstruction and Rendering across Altitudes

TL;DR

The paper tackles the problem of robust 3D reconstruction and novel view synthesis under real-world constraints, such as sparse and heterogeneous imagery from ground, security, and airborne cameras. It introduces a public benchmark dataset with four escalating challenges, centimeter-level geolocation data, and a unified evaluation protocol combining SE$_{90}$ for camera calibration and DreamSim for perceptual view quality, enabling reproducible benchmarking. Baselines built on COLMAP for calibration and 3D Gaussian Splatting via Nerfstudio demonstrate the current state of practice and reveal key research gaps, including cross-view calibration across altitudes, handling visually repetitive scenes, and modeling temporally varying appearance. The dataset and methodology aim to catalyze advances in disaster-response and security applications by providing realistic, multi-altitude data and a framework to foster progress in real-world 3D reconstruction and rendering.

Abstract

Production of photorealistic, navigable 3D site models requires a large volume of carefully collected images that are often unavailable to first responders for disaster relief or law enforcement. Real-world challenges include limited numbers of images, heterogeneous unposed cameras, inconsistent lighting, and extreme viewpoint differences for images collected from varying altitudes. To promote research aimed at addressing these challenges, we have developed the first public benchmark dataset for 3D reconstruction and novel view synthesis based on multiple calibrated ground-level, security-level, and airborne cameras. We present datasets that pose real-world challenges, independently evaluate calibration of unposed cameras and quality of novel rendered views, demonstrate baseline performance using recent state-of-practice methods, and identify challenges for further research.

Figures (9)

• Figure 1: Ground, security, and airborne images are shown for our development benchmark dataset site, illustrating differences in viewpoint and appearance. A 3D point cloud of the full site is shown for context.
• Figure 2: Images from the test datasets from Muscatatuck demonstrate collection at different times of day and year, with varying weather. View synthesis methods must model time-dependent appearance variations to produce accurate rendered images.
• Figure 3: Approximate camera locations for the base challenge datasets are illustrated on a map for the development site at Laurel, Maryland (left) and the test sites at Muscatatuck, Indiana (center and right). Cameras shown green are airborne, red and blue are ground, and others are security. More challenging versions of datasets were produced by reducing image counts for each camera.
• Figure 4: Challenge datasets are produced by defining world coordinates for points or polygons in a scene (top) and then sampling images that observe those points based on normalized distance ($D \in [0,1]$) to image center (bottom).
• Figure 5: DreamSim (labeled DSIM, lower is better) and SSIM (higher is better) are shown for a single reference image compared to novel views rendered from a sequence of 3DGS models. At each step in the sequence, the number of input images for training is reduced in the order of $\{150, 125, 100, 75, 50, 25, 15, 10, 5\}$. DreamSim scores better capture the range of visual similarity.