Audio-Visual Contact Classification for Tree Structures in Agriculture

TL;DR

The paper addresses safe and robust manipulation in cluttered agricultural environments by fusing vibrotactile audio from contact microphones with visual cues to classify tree-contact into leaf, twig, trunk, or ambient. It demonstrates that audio carries rich vibrotactile information while vision provides semantic context, and that their combination yields superior multiclass F1 scores (0.82) and binary F1 scores (0.94) even under cross-embodiment transfer from a handheld probe to a robot. The approach leverages pretrained audio encoders (AST and CLAP) and a ViT-based visual encoder, fused through a lightweight transformer, with training conducted on ~3.5k samples and inference around 14 ms per 1 s segment. A zero-shot transfer capability and an open-source multisensory dataset underscore the practical impact for adaptive, safe manipulation in unstructured agricultural settings. Future work will incorporate temporal modeling to handle sequences and integrate the classification into closed-loop control for real-time manipulation.

Abstract

Contact-rich manipulation tasks in agriculture, such as pruning and harvesting, require robots to physically interact with tree structures to maneuver through cluttered foliage. Identifying whether the robot is contacting rigid or soft materials is critical for the downstream manipulation policy to be safe, yet vision alone is often insufficient due to occlusion and limited viewpoints in this unstructured environment. To address this, we propose a multi-modal classification framework that fuses vibrotactile (audio) and visual inputs to identify the contact class: leaf, twig, trunk, or ambient. Our key insight is that contact-induced vibrations carry material-specific signals, making audio effective for detecting contact events and distinguishing material types, while visual features add complementary semantic cues that support more fine-grained classification. We collect training data using a hand-held sensor probe and demonstrate zero-shot generalization to a robot-mounted probe embodiment, achieving an F1 score of 0.82. These results underscore the potential of audio-visual learning for manipulation in unstructured, contact-rich environments.

Figures (10)

• Figure 1: Robotic arm in an apple orchard. To distinguish safe versus unsafe contacts in a cluttered environment, robot uses multisensory inputs to classify contact events.
• Figure 2: Multi-modal sensor suite in two different embodiments. (Left) Hand-held to facilitate data collection (Right) for robot deployment.
• Figure 3: Audio-image pair of four contact classes. In both sensing modalities, there are distinct patterns between each category that a model can learn from both the spectrograms and images to assist with classification.
• Figure 4: Audio-visual dataset automatically annotated into contact (green) and non-contact (red) intervals using audio-based segmentation. Contact segments are identified by applying a dynamic threshold to the smoothed audio amplitude; values above the threshold are labeled as contact, and those below as non-contact.
• Figure 5: Examples of correct (green) and incorrect (red) predictions from the image-only model. Visual input alone struggles to distinguish subtle differences between near-contact and actual contact, as well as distinguishing ambiguous visual cues in clutter, highlighting the need for complementary haptic sensing.