On-the-fly Feedback SfM: Online Explore-and-Exploit UAV Photogrammetry with Incremental Mesh Quality-Aware Indicator and Predictive Path Planning

TL;DR

The paper tackles real-time UAV photogrammetry by introducing On-the-fly Feedback SfM, an online explore-and-exploit framework that unifies incremental surface reconstruction, mesh-quality assessment, and predictive path planning. It presents an incremental mesh pipeline using dynamic 3D Delaunay triangulation, a tailored energy-based surface extraction, and a dynamic graph-cut solver, paired with ensemble mesh-quality indicators (GSD, redundancy, reprojection error) to guide action. Guided by Q_total, the system performs online low-quality region detection, multi-constraint viewpoint generation, greedy selection, and altitude-aware trajectory optimization to adapt flight in real time. Extensive experiments on self-captured and public datasets demonstrate competitive mesh quality, robust online performance, and significant reflight reduction, illustrating the practical potential for disaster response and digital-twin maintenance. The approach delivers a closed-loop acquisition–reconstruction–feedback–planning cycle that shifts UAV photogrammetry from passive data collection to intelligent, adaptive sensing.

Abstract

Compared with conventional offline UAV photogrammetry, real-time UAV photogrammetry is essential for time-critical geospatial applications such as disaster response and active digital-twin maintenance. However, most existing methods focus on processing captured images or sequential frames in real time, without explicitly evaluating the quality of the on-the-go 3D reconstruction or providing guided feedback to enhance image acquisition in the target area. This work presents On-the-fly Feedback SfM, an explore-and-exploit framework for real-time UAV photogrammetry, enabling iterative exploration of unseen regions and exploitation of already observed and reconstructed areas in near real time. Built upon SfM on-the-fly , the proposed method integrates three modules: (1) online incremental coarse-mesh generation for dynamically expanding sparse 3D point cloud; (2) online mesh quality assessment with actionable indicators; and (3) predictive path planning for on-the-fly trajectory refinement. Comprehensive experiments demonstrate that our method achieves in-situ reconstruction and evaluation in near real time while providing actionable feedback that markedly reduces coverage gaps and re-flight costs. Via the integration of data collection, processing, 3D reconstruction and assessment, and online feedback, our on the-fly feedback SfM could be an alternative for the transition from traditional passive working mode to a more intelligent and adaptive exploration workflow. Code is now available at https://github.com/IRIS-LAB-whu/OntheflySfMFeedback.

Figures (14)

• Figure 1: The overall workflow of our On-the-fly Feedback SfM. Each new batch of UAV images are incrementally processed via SfM on-the-fly to refine camera poses and update the sparse point cloud. The evolving sparse cloud is immediately converted into a triangular surface via the incremental surface reconstruction module (yellow panel). This process involves updating the dynamic 3D Delaunay triangulation, accumulating connectivity weights via ray casting, and applying dynamic graph-cut to obtain the optimal partition of tetrahedra into inside and outside space. Online quality assessment computes per-face photogrammetric indicators and identifies low-quality regions, which guide multi-constraint viewpoint generation and real-time trajectory optimization, the resulting trajectory segment is then executed as new fly path by the UAV before the next batch of images arrives.
• Figure 2: Illustration of the Energy Function model. A line of sight from a camera to an observed point distributes weights to the originating tetrahedron, the intersected facets, and the terminating tetrahedron.
• Figure 3: Comparative results of the Mesh Triangle Outlier Filtering method, (a) Original mesh; (b) $n_{it}=5$; (c) $n_{it}=10$; (d) $n_{it}=15$. Highlighted (red) regions denote the abnormal triangles removed during corresponding iterations. (e) shows mesh filtering statistics in each iteration.
• Figure 4: Workflow of the proposed predictive path planning pipeline. It contains five steps: (a) Low-quality regions are identified based on the per-face ensemble quality score $Q_{total}$. (b) DBSCAN clustering groups spatially adjacent low-quality faces into coherent regions, with distinct colors indicating different clusters. (c) Candidate viewpoints are generated around each region under multi-constraint sampling. (d) A sparsification step selects a compact, well-distributed subset of viewpoints. (e) The final trajectory is optimized for smooth and efficient UAV flight.
• Figure 5: Sample images of experimental datasets.