RoMu4o: A Robotic Manipulation Unit For Orchard Operations Automating Proximal Hyperspectral Leaf Sensing

TL;DR

RoMu4o advances autonomous proximal hyperspectral leaf sensing in orchard environments by integrating a 6-DOF robotic manipulator, RGB-D perception, an integrated hyperspectral sensor with independent illumination, and a DL-based leaf segmentation and 6D pose estimation pipeline. The system couples a Jacobian-based IK solver with a rapid RRTC-like path planner to enable collision-aware leaf grasping and spectral data collection, validated in lab magnolia and field pistachio settings. Key results show 1-LPB hyperspectral sampling at 95% in-lab and 79% in field, with an overall leaf-grasping success of 66% across scenarios and 70% in the field, demonstrating practical viability in unstructured orchard environments. The work contributes a complete hardware–software stack, including open-source code, and demonstrates the potential to scale high-fidelity hyperspectral phenotyping in commercial orchards through automated data collection and robust perception–manipulation integration.

Abstract

Driven by the need to address labor shortages and meet the demands of a rapidly growing population, robotic automation has become a critical component in precision agriculture. Leaf-level hyperspectral spectroscopy is shown to be a powerful tool for phenotyping, monitoring crop health, identifying essential nutrients within plants as well as detecting diseases and water stress. This work introduces RoMu4o, a robotic manipulation unit for orchard operations offering an automated solution for proximal hyperspectral leaf sensing. This ground robot is equipped with a 6DOF robotic arm and vision system for real-time deep learning-based image processing and motion planning. We developed robust perception and manipulation pipelines that enable the robot to successfully grasp target leaves and perform spectroscopy. These frameworks operate synergistically to identify and extract the 3D structure of leaves from an observed batch of foliage, propose 6D poses, and generate collision-free constraint-aware paths for precise leaf manipulation. The end-effector of the arm features a compact design that integrates an independent lighting source with a hyperspectral sensor, enabling high-fidelity data acquisition while streamlining the calibration process for accurate measurements. Our ground robot is engineered to operate in unstructured orchard environments. However, the performance of the system is evaluated in both indoor and outdoor plant models. The system demonstrated reliable performance for 1-LPB hyperspectral sampling, achieving 95% success rate in lab trials and 79% in field trials. Field experiments revealed an overall success rate of 70% for autonomous leaf grasping and hyperspectral measurement in a pistachio orchard. The open-source repository is available at: https://github.com/mehradmrt/UCM-AgBot-ROS2

Figures (5)

• Figure 1: (a) RoMu4o, our proposed robotic manipulation unit for orchard operations aimed for autonomous proximal hyperspectral leaf sensing. The system is comprised of 1) 6-DOF robotic manipulator 2) RGBD camera 3) two-finger gripper end-effector integrated with a hyperspectral sensing system 4) an independent VIS-NIR light source for referencing and self-calibration 5) crawler-type ground robot 6) electric and control boxes, hardware drivers, portable batteries, processing unit, Ethernet hub, and auxiliary hardware needed for the UGV's navigation. (b) Pistachio leaves used for field experiments; (c) Magnolia leaves used for lab experiments.
• Figure 2: Flowchart for the autonomous proximal hyperspectral leaf sensing process. The workflow consists of a perception pipeline for image processing and 6D pose estimation followed by a robotic manipulation pipeline for precise leaf grasping and hyperspectral data collection.
• Figure 3: Perception pipeline for leaf manipulation (a) Field and (b) lab experiments showing (i) the RGB image as seen by the RealSense depth camera; (ii) leaf detection and segmentation using pre-trained instance segmentation deep-learning networks; (iii) extracting the 3d structure of identified target leaves by aligning the ordered depth channel to RGB pixels; and (iv) noise elimination and 6D pose estimation for each target.
• Figure 4: Leaf manipulation process in action. (a) Pistachio trees representing field experiments on the top row; (b) magnolia shrub representing lab experiments on the bottom row. (i) a batch of foliage is observed by the camera for the perception pipeline to generate 6D poses for leaf targets; (ii) pose estimates are processed through the robotic manipulation workflow for path planning and trajectory execution while the hyperspectral sensor is calibrated; (iii) the first target is approached and grasped, hyperspectral data collected and stored; (iv) planning to the next target for leaf grasping and spectroscopy measurements.
• Figure 5: Leaf hyperspectral data collected using our proposed robotic system; (a) Spectral transmittance of magnolia leaves used as a plant model for indoor lab experiments; (b) Spectral transmittance of pistachio leaves representing our outdoor field experiments.