Iterative Motion Compensation for Canonical 3D Reconstruction from UAV Plant Images Captured in Windy Conditions

TL;DR

This work tackles robust 3D reconstruction of individual plants from UAV imagery captured in wind, where leaf motion degrades traditional methods. It introduces a general, iterative motion-compensation pipeline that warps input images toward a motion-free canonical frame by rendering intermediate reconstructions and estimating optical flow with RAFT; this can be combined with any baseline 3D reconstruction method, such as 3D Gaussian Splatting or deformable variants. The approach yields sharper, high-resolution meshes and improved perceptual metrics, validated on a dataset of plants across crops and growth stages, with substantial gains evident in early iterations. By enabling reliable canonical plant meshes from affordable UAV data, the method has practical impact for high-throughput phenotyping and crop monitoring, while remaining extensible to future mesh-extraction improvements and faster training.

Abstract

3D phenotyping of plants plays a crucial role for understanding plant growth, yield prediction, and disease control. We present a pipeline capable of generating high-quality 3D reconstructions of individual agricultural plants. To acquire data, a small commercially available UAV captures images of a selected plant. Apart from placing ArUco markers, the entire image acquisition process is fully autonomous, controlled by a self-developed Android application running on the drone's controller. The reconstruction task is particularly challenging due to environmental wind and downwash of the UAV. Our proposed pipeline supports the integration of arbitrary state-of-the-art 3D reconstruction methods. To mitigate errors caused by leaf motion during image capture, we use an iterative method that gradually adjusts the input images through deformation. Motion is estimated using optical flow between the original input images and intermediate 3D reconstructions rendered from the corresponding viewpoints. This alignment gradually reduces scene motion, resulting in a canonical representation. After a few iterations, our pipeline improves the reconstruction of state-of-the-art methods and enables the extraction of high-resolution 3D meshes. We will publicly release the source code of our reconstruction pipeline. Additionally, we provide a dataset consisting of multiple plants from various crops, captured across different points in time.

Figures (8)

• Figure 1: a): Aligned (dense) scene reconstruction using COLMAP schoenberger2016sfm, including the estimated camera poses. b): Textured mesh extracted after 100 iterations with our proposed method (3D Gaussian Splatting kerbl20233d was used as the baseline).
• Figure 2: Pipeline of the autonomous capturing method. Firstly, visual data together with UAV orientation, altitude, velocity, and gimbal orientation are received from the UAV. After that, visible markers are extracted, and the UAV position is approximated from them. Data from the UAV, together with the approximated position from markers and the previous command, is sent to the Kalman filter. The current position is approximated and passed to the path planner, which creates a trajectory through desired waypoints. Finally, the motion controller receives the trajectory and sends the velocity command back to the UAV.
• Figure 3: Marker sets with examples of detected markers.
• Figure 4: Canonical 3D Reconstruction Pipeline. In the first iteration, the original input images are used to perform 3D Gaussian Splatting kerbl20233d. Subsequently, the input views are then rendered with the same camera parameters (Predictions). Using the predictions and original images, we then estimate the optical flow from the predictions to the original input images, which is then used to deform the input images into the current predictions (Deformed Images). In the following iterations, the 3D Gaussian Splatting is performed using the deformed images instead. These steps are repeated for a predefined number of iterations. Images cropped for visualization.
• Figure 5: Comparison of 3D Gaussian Splatting kerbl20233d (GS) and our proposed method with optical flow compensation. The first row (GS) shows results from standard 3D Gaussian Splatting without motion compensation. The second and third rows (+Ours-30 and +Ours-100) show our results after 30 and 100 iterations of optical flow compensation, respectively. The fourth and fifth rows (GS Mesh and +Ours-100 Mesh) show meshes extracted from the 2D Gaussian representation before and after compensation. The last row (GT) presents the ground truth images for reference. Areas with notable improvements in visual quality are highlighted in the zoom-in boxes. Our approach leads to visibly sharper and more consistent textures across the scene, especially in regions affected by wind.