T-REX: Vision-Based System for Autonomous Leaf Detection and Grasp Estimation

TL;DR

Automating leaf tissue sampling for in-situ plant pathogen detection is the problem addressed. The authors implement T-Rex, a 6-DOF gantry robot that integrates a stereo vision pipeline (YOLOv8 for leaf segmentation and RAFT-Stereo for depth) with a geometry-driven grasp planner and OMPL/MoveIt-based motion control to autonomously locate, grasp, and sample leaves using a microneedle array. Key contributions include the end-effector design, a Pareto-based leaf and grasp-point selection scheme, a robust open-source-like ROS-based control architecture, and an automated microneedle reload mechanism enabling repeated sampling. Experimental validation on artificial plant models yields a grasp success rate of 66.6% across 24 trials, demonstrating the feasibility of automated leaf sampling and DNA extraction workflows in controlled environments; the work lays a foundation for scalable plant health surveillance in CEA.

Abstract

T-Rex (The Robot for Extracting Leaf Samples) is a gantry-based robotic system developed for autonomous leaf localization, selection, and grasping in greenhouse environments. The system integrates a 6-degree-of-freedom manipulator with a stereo vision pipeline to identify and interact with target leaves. YOLOv8 is used for real-time leaf segmentation, and RAFT-Stereo provides dense depth maps, allowing the reconstruction of 3D leaf masks. These observations are processed through a leaf grasping algorithm that selects the optimal leaf based on clutter, visibility, and distance, and determines a grasp point by analyzing local surface flatness, top-down approachability, and margin from edges. The selected grasp point guides a trajectory executed by ROS-based motion controllers, driving a custom microneedle-equipped end-effector to clamp the leaf and simulate tissue sampling. Experiments conducted with artificial plants under varied poses demonstrate that the T-Rex system can consistently detect, plan, and perform physical interactions with plant-like targets, achieving a grasp success rate of 66.6\%. This paper presents the system architecture, implementation, and testing of T-Rex as a step toward plant sampling automation in Controlled Environment Agriculture (CEA).

Figures (21)

• Figure 1: T-REX system architecture showing the main processing pipeline: initial stereo image acquisition, perception and leaf grasping analysis, motion planning for execution, leaf grasping operations, and the reload mechanism for continuous operation. The feedback loops (dotted lines) enable autonomous multi-sample collection without human intervention.
• Figure 2: CAD rendering of T-Rex’s wrist and end-effector subsystem. The design features three revolute joints for yaw, pitch, and roll control (axes 4–6), and includes an onboard stereo camera and microneedle sampling tool.
• Figure 3: The T-Rex gantry robot setup inside a controlled lab environment. It spans a 3m $\times$ 1.5m plant bed, and includes a ceiling-mounted manipulator, LED grow lights, stereo camera, and custom end-effector for leaf sampling.
• Figure 4: YOLOv8 leaf segmentation pipeline. Left: Original top-down image of a tomato plant tray. Center: Segmentation mask output, where each color represents a unique leaf ID. Right: Visualization overlay showing confidence scores assigned to each detected leaf.
• Figure 5: T-Rex system block diagram showing the complete pipeline from stereo image acquisition through vision processing, grasp point selection, motion planning, and reload mechanism.