Self-Supervised Learning for Robotic Leaf Manipulation: A Hybrid Geometric-Neural Approach

TL;DR

This work tackles autonomous leaf manipulation amid deformable, occluded plant leaves by presenting a hybrid geometric-neural framework that fuses YOLOv8-based segmentation and RAFT-Stereo depth with a self-supervised training regime. A geometric feature scoring pipeline provides interpretable, constraint-driven grasp candidates, while GraspPointCNN offers learned refinement guided by a confidence measure, with predictions integrated through a dynamic, confidence-weighted fusion. The approach achieves 88.0% success in controlled settings and 84.7% in real greenhouses, substantially outperforming purely geometric or neural baselines and enabling continuous improvement through ongoing data collection without manual annotations. The results demonstrate the practical viability of combining model-driven and data-driven methods for robust agricultural robotics under variable morphologies and environments, laying the groundwork for scalable automated crop monitoring and manipulation systems.

Abstract

Automating leaf manipulation in agricultural settings faces significant challenges, including the variability of plant morphologies and deformable leaves. We propose a novel hybrid geometric-neural approach for autonomous leaf grasping that combines traditional computer vision with neural networks through self-supervised learning. Our method integrates YOLOv8 for instance segmentation and RAFT-Stereo for 3D depth estimation to build rich leaf representations, which feed into both a geometric feature scoring pipeline and a neural refinement module (GraspPointCNN). The key innovation is our confidence-weighted fusion mechanism that dynamically balances the contribution of each approach based on prediction certainty. Our self-supervised framework uses the geometric pipeline as an expert teacher to automatically generate training data. Experiments demonstrate that our approach achieves an 88.0% success rate in controlled environments and 84.7% in real greenhouse conditions, significantly outperforming both purely geometric (75.3%) and neural (60.2%) methods. This work establishes a new paradigm for agricultural robotics where domain expertise is seamlessly integrated with machine learning capabilities, providing a foundation for fully automated crop monitoring systems.

Figures (13)

• Figure 1: System architecture showing the integration of vision pipeline, grasp point selection, and robot manipulation modules. The hybrid approach combines geometric feature scoring with neural refinement through confidence-weighted fusion.
• Figure 2: Vision pipeline outputs: (a) Instance segmentation with individual leaf masks, (b) RAFT-Stereo disparity map, (c) 3D point cloud reconstruction with highlighted target leaf.
• Figure 3: RAFT-Stereo outputs showing the processing pipeline: (a) Raw image from the left camera of the stereo pair, (b) Generated disparity map where warmer colors indicate closer objects, (c) Final 3D reconstruction combining depth and segmentation data.
• Figure 4: Signed Distance Field (SDF) visualization for grasp planning: (a) Raw plant image with leaf candidates, (b) SDF representation showing free space (purple/blue) and occupied regions (yellow/red). Red rays indicate potential grasp approach directions.
• Figure 6: CAD rendering of T-Rex's wrist and end-effector subsystem. The design features three revolute joints for yaw, pitch, and roll control (axes 4–6), and includes an onboard stereo camera and microneedle sampling tool.