HortiBot: An Adaptive Multi-Arm System for Robotic Horticulture of Sweet Peppers

TL;DR

HortiBot addresses the automation gap in selective harvesting within semi-structured greenhouses by introducing a three-arm platform that combines active perception, online world modeling, and collision-aware manipulation. The approach integrates a perception pipeline (peppers, peduncles, stems) with 3D mapping (Voxblox++) and adaptive manipulation (PMP and OTG) to perform robust, force-sensing-guided harvesting. Key contributions include a comprehensive hardware/workspace analysis, a novel peduncle detection strategy, online perception during manipulation, and a dual-arm manipulation framework validated in indoor pepper plant mock-ups achieving 83% overall success and 27 s per fruit. The work demonstrates a viable end-to-end solution toward autonomous selective harvesting and positive implications for deployment in glasshouse environments.

Abstract

Horticultural tasks such as pruning and selective harvesting are labor intensive and horticultural staff are hard to find. Automating these tasks is challenging due to the semi-structured greenhouse workspaces, changing environmental conditions such as lighting, dense plant growth with many occlusions, and the need for gentle manipulation of non-rigid plant organs. In this work, we present the three-armed system HortiBot, with two arms for manipulation and a third arm as an articulated head for active perception using stereo cameras. Its perception system detects not only peppers, but also peduncles and stems in real time, and performs online data association to build a world model of pepper plants. Collision-aware online trajectory generation allows all three arms to safely track their respective targets for observation, grasping, and cutting. We integrated perception and manipulation to perform selective harvesting of peppers and evaluated the system in lab experiments. Using active perception coupled with end-effector force torque sensing for compliant manipulation, HortiBot achieves high success rates in our indoor pepper plant mock-up.

Figures (12)

• Figure 1: HortiBot: A three-arm system with active perception and dual-arm manipulation for robotic horticulture. The right arm is used for grasping, the left arm performs cutting, and the central arm moves stereo cameras for mapping and online observation.
• Figure 2: HortiBot hardware setup.
• Figure 3: Workflow for autonomous selective pepper harvesting. Colors depict different actions: (see Sec. \ref{['sec:perception']}), (see Sec. \ref{['sec:auto_behaviour']}), manipulation using (see Sec. \ref{['sec:pmp']}), and (see Sec. \ref{['sec:otg']}).
• Figure 4: Workspace analysis: Reachable (green) and non-reachable (yellow) fruit poses for both manipulation arms in the selected arm configuration.
• Figure 5: Cropped peduncle detection. Pepper and cropped peduncle detection applied on the fruits in the Campus Klein Altendorf glasshouse pepper plants. As can been seen, there are multiple peppers with peduncles not easily identifiable in the full image. The image on the right shows the cropped image obtained by inflating the pepper's bounding box and the resultant pepper and peduncle detected.