Creating a Segmented Pointcloud of Grapevines by Combining Multiple Viewpoints Through Visual Odometry

TL;DR

Problem: automating pruning-point detection in grapevines is difficult due to occlusions and morphological variability. Approach: fuse multiple viewpoints by combining 2D segmentation with 3D pointcloud reconstruction to produce a segmented 4D representation for pruning. Key contributions: (1) detectron2-based segmentation of five organs, (2) robust keypoint matching with SuperPoint and LightGlue, (3) 3D registration by Orthogonal Procrustes and pose-graph optimization, (4) HDBSCAN-based clustering to unite views. Significance: enables fast, robot-assisted pruning with precise spatial and semantic information to guide pruning decisions.

Abstract

Grapevine winter pruning is a labor-intensive and repetitive process that significantly influences the quality and quantity of the grape harvest and produced wine of the following season. It requires a careful and expert detection of the point to be cut. Because of its complexity, repetitive nature and time constraint, the task requires skilled labor that needs to be trained. This extended abstract presents the computer vision pipeline employed in project Vinum, using detectron2 as a segmentation network and keypoint visual odometry to merge different observation into a single pointcloud used to make informed pruning decisions.

Figures (3)

• Figure 1: Output inference of detectron2, giving precise segmentation labels of the five classes.
• Figure 2: Keypoint detection and matching using Superpoint and Lightglue.
• Figure 3: Final pointcloud with segmented organs.