Hierarchical Tri-manual Planning for Vision-assisted Fruit Harvesting with Quadrupedal Robots

TL;DR

The paper tackles autonomous fruit harvesting in unstructured outdoor environments using a three-arm quadrupedal robot. It introduces LocoHarv-3, a three-arm platform, and a hierarchical tri-manual planning framework combining Next-Best View planning for the Spot arm with a constrained quadratic program and analytic forward kinematics for the dual arms, all integrated with learning-based perception and LiDAR SLAM. Key results show a 90% indoor success rate on single attempts and 40% outdoor success, validating robustness and highlighting outdoor challenges such as perception noise and wind. The work provides a path toward scalable autonomous harvesting across crops and terrains and suggests future directions including broader crops, LiDAR-enhanced pose estimation, multi-modal harvesting, and soft-robotics-enabled end-effectors.

Abstract

This paper addresses the challenge of developing a multi-arm quadrupedal robot capable of efficiently harvesting fruit in complex, natural environments. To overcome the inherent limitations of traditional bimanual manipulation, we introduce the first three-arm quadrupedal robot LocoHarv-3 and propose a novel hierarchical tri-manual planning approach, enabling automated fruit harvesting with collision-free trajectories. Our comprehensive semi-autonomous framework integrates teleoperation, supported by LiDAR-based odometry and mapping, with learning-based visual perception for accurate fruit detection and pose estimation. Validation is conducted through a series of controlled indoor experiments using motion capture and extensive field tests in natural settings. Results demonstrate a 90\% success rate in in-lab settings with a single attempt, and field trials further verify the system's robustness and efficiency in more challenging real-world environments.

Figures (8)

• Figure 1: Illustration of semi-autonomous avocado harvesting using the proposed LocoHarv-3 platform and tri-manual path planning.
• Figure 2: The quadrupedal tri-manual platform LocoHarv-3.
• Figure 3: Overview of our hierarchical tri-manual planning approach.
• Figure 4: Workflow of the learning-based fruit detection and pose estimation using an RGB-D camera.
• Figure 5: Flowchart of the semi-autonomous fruit harvesting framework with the hierarchical tri-manual planning, learning-based visual perception, and LiDAR-SLAM-assisted teleoperation.