From Orbit to Ground: Generative City Photogrammetry from Extreme Off-Nadir Satellite Images

TL;DR

This work tackles city-scale 3D reconstruction from extreme off-nadir satellite imagery, where large viewpoint gaps and texture blur hinder existing NeRF/3DGS pipelines. It introduces a two-stage framework: a robust geometry stage using a Z-Monotonic SDF to produce a watertight 2.5D city mesh, followed by a diffusion-based, deterministic texture restoration network for high-frequency appearance; the final textures are refined iteratively. The method achieves state-of-the-art geometric and visual quality on synthetic and real-world datasets, including a 4 km^2 real region from sparse satellite views, enabling photorealistic ground-view synthesis and downstream urban applications. It offers a scalable, asset-ready approach for urban digital twins, with limitations on non-monotonic structures and potential over-smoothing of unique textures.

Abstract

City-scale 3D reconstruction from satellite imagery presents the challenge of extreme viewpoint extrapolation, where our goal is to synthesize ground-level novel views from sparse orbital images with minimal parallax. This requires inferring nearly $90^\circ$ viewpoint gaps from image sources with severely foreshortened facades and flawed textures, causing state-of-the-art reconstruction engines such as NeRF and 3DGS to fail. To address this problem, we propose two design choices tailored for city structures and satellite inputs. First, we model city geometry as a 2.5D height map, implemented as a Z-monotonic signed distance field (SDF) that matches urban building layouts from top-down viewpoints. This stabilizes geometry optimization under sparse, off-nadir satellite views and yields a watertight mesh with crisp roofs and clean, vertically extruded facades. Second, we paint the mesh appearance from satellite images via differentiable rendering techniques. While the satellite inputs may contain long-range, blurry captures, we further train a generative texture restoration network to enhance the appearance, recovering high-frequency, plausible texture details from degraded inputs. Our method's scalability and robustness are demonstrated through extensive experiments on large-scale urban reconstruction. For example, in our teaser figure, we reconstruct a $4\,\mathrm{km}^2$ real-world region from only a few satellite images, achieving state-of-the-art performance in synthesizing photorealistic ground views. The resulting models are not only visually compelling but also serve as high-fidelity, application-ready assets for downstream tasks like urban planning and simulation. Project page can be found at https://pku-vcl-geometry.github.io/Orbit2Ground/.

Figures (27)

• Figure 1: City-Scale 3D Reconstruction from Satellite Imagery. We reconstruct a $4\,\mathrm{km}^2$ real-world urban region from 11 sparse-view satellite images captured from orbit that contain extremely limited parallax. The resulting 3D model, featuring crisp geometry and photorealistic appearance, enables extreme viewpoint extrapolation, supporting high-fidelity, close-range rendering from ground-level viewpoints. Please zoom in for details.
• Figure 2: Unlike dense street views, satellite images are sparse and captured with extreme off-nadir angles. This leads to a severe deficiency in parallax for vertical structures. Yellow points represent 3D locations determined by MVS, whereas satellite images only recover ground and roof surfaces.
• Figure 3: The framework of our method. Our pipeline first reconstructs city geometry, then refines its appearance.Stage 1 (Geometry): We optimize a Z-Monotonic SDF against sparse MVS points to extract a high-fidelity, watertight mesh with clean vertical facades. Stage 2 (Appearance): Starting with an initial texture (back-projected from source images), we use a restoration network to enhance close-range novel-view renderings, which further serve as sharp, high-fidelity supervision for final texture optimization.
• Figure 4: Z-Monotonic SDF vs. Naive Conversion. (a, b) A naive 2.5D mesh, generated by directly converting sparse MVS points into a voxel grid, suffers from severe "stair-step" artifacts and topological holes. (c) Our Z-Monotonic SDF representation optimizes a continuous field, resulting in a clean, watertight mesh with precise roofs and sharp vertical facades.
• Figure 5: Appearance Refinement. (a) The basic texture $\mathbf{T}_\text{basic}$, created by naively back-projecting the blurry source satellite images, suffers from low fidelity and "baked-in" artifacts. (b) Our final texture $\mathbf{T}_\text{final}$, optimized using supervision from the restoration network, recovers sharp, photorealistic, and globally consistent details.