A real-time, hardware agnostic framework for close-up branch reconstruction using RGB data

TL;DR

The paper tackles the challenge of real-time, close-up 3D tree modeling for pruning using only an RGB camera. It introduces a real-time, hardware-agnostic framework that actively scans along a primary branch to reconstruct a 3D centerline and radius for both the primary and attached secondary branches, using a four-node system (segmentation, pixel tracking/triangulation, 3D modeling, and a branch-following controller). The reconstruction relies on 2D Bézier curve fitting lifted to 3D via PIPs triangulation, combined with a 3D skeleton maintained along the branch, and an active vision controller that rotates the view to reveal occluded branches. Key findings show high accuracy in simulation (PB residuals ~3–4 mm, SB orientation ~±15°) and real-time performance (up to ~10 cm/s) with real-world challenges in lab and orchard settings, where segmentation quality and scene clutter affect accuracy. The work demonstrates that a lightweight, RGB-only approach with active scanning can produce actionable pruning models, reducing dependence on stereo or depth sensors for in-field pruning tasks.

Abstract

Creating accurate 3D models of tree topology is an important task for tree pruning. The 3D model is used to decide which branches to prune and then to execute the pruning cuts. Previous methods for creating 3D tree models have typically relied on point clouds, which are often computationally expensive to process and can suffer from data defects, especially with thin branches. In this paper, we propose a method for actively scanning along a primary tree branch, detecting secondary branches to be pruned, and reconstructing their 3D geometry using just an RGB camera mounted on a robot arm. We experimentally validate that our setup is able to produce primary branch models with 4-5 mm accuracy and secondary branch models with 15 degrees orientation accuracy with respect to the ground truth model. Our framework is real-time and can run up to 10 cm/s with no loss in model accuracy or ability to detect secondary branches.

Figures (7)

• Figure 1: Our framework uses 2D RGB data and knowledge of the robot's kinematics, camera intrinsics, and hand-eye calibration to follow along a primary branch and create highly accurate 3D branch reconstructions of the primary and secondary branches that can be used immediately for picking a cut and executing it.
• Figure 2: A system overview of our framework for branch scanning and reconstruction. It consists of 4 system nodes: A segmentation node producing binary masks; a point tracking plus triangulation node that reads in RGB images and returns 3D estimates for queried 2D pixels in an image; the core node that performs the branch reconstruction; and the controller node that moves the robot along the 3D primary branch. Dotted lines represent asynchronous system inputs.
• Figure 3: The process for extracting 2D primary and secondary branches from a skeleton. We start with the medial axis skeleton and find the subpath with the best fit to be the main branch. We then check skeletal paths extending off the primary branch to see if they are sufficiently long and a Bézier curve can be fit to them. We subsample the curves to identify pixels for which we will estimate 3D positions.
• Figure 4: Using the Persistent Independent Particles point tracker, along with knowledge of the camera's poses and intrinsics, we can obtain 3D estimates of an arbitrary subset of pixels in an image.
• Figure 5: Components of the primary branch following controller. The controller follows the gradient of the 3D centerline that lies in front of the camera, with velocity offsets to keep a fixed distance from the branch and to keep the branch centered in the image (top-right). To gain a view of potentially occluded branches, it also periodically cycles through 3 camera orientations at an angle $\theta$ (bottom-right, shown from a top view).