Safe Leaf Manipulation for Accurate Shape and Pose Estimation of Occluded Fruits

TL;DR

This work tackles the challenge of occlusion in automated fruit monitoring by enabling a robot to physically manipulate occluding leaves. It introduces scene-consistent shape completion (SSC) that leverages semantic priors to jointly complete fruit and branch geometry, a perception-based deformation model to predict leaf responses, and Leaf Manipulation Action Planning (LMAP) to select safe, visibility-enhancing actions. The approach is validated on artificial and real sweet pepper plants, showing superior shape/pose estimation under heavy occlusion and safer, more effective occlusion removal compared with baselines, including in multi-fruit scenes. The results suggest significant practical impact for autonomous crop monitoring in greenhouse environments, with future work aiming at glasshouse deployment and multi-arm coordination for broader viewpoint planning.

Abstract

Fruit monitoring plays an important role in crop management, and rising global fruit consumption combined with labor shortages necessitates automated monitoring with robots. However, occlusions from plant foliage often hinder accurate shape and pose estimation. Therefore, we propose an active fruit shape and pose estimation method that physically manipulates occluding leaves to reveal hidden fruits. This paper introduces a framework that plans robot actions to maximize visibility and minimize leaf damage. We developed a novel scene-consistent shape completion technique to improve fruit estimation under heavy occlusion and utilize a perception-driven deformation graph model to predict leaf deformation during planning. Experiments on artificial and real sweet pepper plants demonstrate that our method enables robots to safely move leaves aside, exposing fruits for accurate shape and pose estimation, outperforming baseline methods. Project page: https://shaoxiongyao.github.io/lmap-ssc/.

Figures (7)

• Figure 1: Example of our system in a setting with sweet peppers. Left: The zoomed-in area in the initial observation showcases the occlusion of the targeted pepper. The red profiles represent different possible shapes and poses of the fruit consistent with the visible portion. Right: Our method selects an appropriate manipulation action to reveal the entire fruit while preserving the leaf's integrity. This enhanced visibility enables accurate shape and pose estimation.
• Figure 2: Overview of the proposed active fruit shape and pose estimation system for plant-safe occlusion removal. Given an RGB-D image, a semantic representation informs our scene-consistent shape completion module to build an initial estimate of the fruit shape and pose. The deformation prediction is performed on the same semantic representation for acquiring the deformed states of the leaf, while the action planner predicts fruit visibility and damage risk for several candidate actions, and the best manipulation is executed. After execution, the fruit's shape and pose are re-estimated using the updated observation information.
• Figure 3: Representative leaf occlusion scenarios in the experiments: (a) and (b) are examples of artificial leaf and pepper configurations used in Sec. \ref{['sec:EvalShapeCompletion']} and \ref{['sec:EvalActionPlanning']}, respectively; (c) is an example of the real pepper leaf and real pepper configuration used in Sec. \ref{['sec:EvalRealPlant']}.
• Figure 4: Illustration of shape completion with initial and without leaf occlusion. Red mesh is the ground truth fruit and blue mesh indicates the estimated fruit with different methods. Red/blue lines indicate ground truth/estimated peduncle axes.
• Figure 5: Illustration of (a) the initial scenario, (b, d-f) different leaf manipulation actions, and (c) the final estimation of our method.