Active Semantic Mapping with Mobile Manipulator in Horticultural Environments

TL;DR

This work tackles the challenge of building informative semantic maps in agricultural settings where occlusions and varying conditions hinder passive sensing. It presents a four-module system that combines a Semantic Extractor, a Semantic Octomap-based Mapping module, a cluster-based Viewpoint Planner, and a Planning and Control component, augmented by a novel OSAMCEP information gain metric that accounts for occlusions and multiple semantic classes. Through simulation and real-world tests, the approach achieves higher fruit surface coverage and lower multi-class entropy than baselines, while offering a 8% reduction in NBV planning time relative to frontier-based methods, at the cost of increased overall runtime due to motion planning. The work advances target-aware active semantic mapping in horticulture and points to future gains from integrating advanced reconstruction techniques to further mitigate depth and segmentation noise in dynamic environments.

Abstract

Semantic maps are fundamental for robotics tasks such as navigation and manipulation. They also enable yield prediction and phenotyping in agricultural settings. In this paper, we introduce an efficient and scalable approach for active semantic mapping in horticultural environments, employing a mobile robot manipulator equipped with an RGB-D camera. Our method leverages probabilistic semantic maps to detect semantic targets, generate candidate viewpoints, and compute corresponding information gain. We present an efficient ray-casting strategy and a novel information utility function that accounts for both semantics and occlusions. The proposed approach reduces total runtime by 8% compared to previous baselines. Furthermore, our information metric surpasses other metrics in reducing multi-class entropy and improving surface coverage, particularly in the presence of segmentation noise. Real-world experiments validate our method's effectiveness but also reveal challenges such as depth sensor noise and varying environmental conditions, requiring further research.

Figures (7)

• Figure 1: Top: Simulation environment with capsicum plants and Husky Robot. Bottom left: Reconstructions done using our approach. Bottom right: Reconstruction done using predefined dense scanning. Blue circles highlight incomplete fruit areas.
• Figure 2: System overview. Semantic extractor: semantic segmentation is used to extract fruits, leaves, and background in the scene. Mapping: a 3D semantic and probabilistic representation of the environment is generated. Viewpoint planner: It generates potential viewpoint candidates and chooses those with the highest information gain. Planning and control: collision-free trajectories to the desired goals and control signals are generated.
• Figure 3: Total entropy and surface coverage with our active mapping approach, a frontier-based approach, and predefined scanning, with (dashed lines) and without (solid lines) segmentation noise. All the methods execute 120 viewpoints along the path between two crop rows with 8 plants in total.
• Figure 4: Runtime comparison between our approach and two baselines. The total time is divided into three tasks: mapping, NBV planning, and motion planning and control. Note that our approach significantly reduces the NBV planning time compared to the frontier-based method.
• Figure 5: Entropy and surface coverage for different downsampling strategies. Solid lines and dashed lines indicate viewpoints taken at 0.4 m and 0.6 m from cluster centroids, respectively. Note that despite the distance from the targets, our downsampling strategy maintains close performance to dense sampling.