ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Data

TL;DR

ManiWAV is introduced: an 'ear-in-hand' data collection device to collect in-the-wild human demonstrations with synchronous audio and visual feedback, and a corresponding policy interface to learn robot manipulation policy directly from the demonstrations.

Abstract

Audio signals provide rich information for the robot interaction and object properties through contact. This information can surprisingly ease the learning of contact-rich robot manipulation skills, especially when the visual information alone is ambiguous or incomplete. However, the usage of audio data in robot manipulation has been constrained to teleoperated demonstrations collected by either attaching a microphone to the robot or object, which significantly limits its usage in robot learning pipelines. In this work, we introduce ManiWAV: an 'ear-in-hand' data collection device to collect in-the-wild human demonstrations with synchronous audio and visual feedback, and a corresponding policy interface to learn robot manipulation policy directly from the demonstrations. We demonstrate the capabilities of our system through four contact-rich manipulation tasks that require either passively sensing the contact events and modes, or actively sensing the object surface materials and states. In addition, we show that our system can generalize to unseen in-the-wild environments by learning from diverse in-the-wild human demonstrations.

Figures (13)

• Figure 1: Contact sound reveals rich information. More specifically, (a) contact events, such as eraser touching the whiteboard; (b) different contact modes, such as spatula poking on the side of the bagel versus sliding below the bagel; (c) surface material, such as the furry ('loop') and rough ('hook') side of velcro tapes; (d) object states, such as whether the cup contains objects or not.
• Figure 2: Ear-in-hand gripper for in-the-wild data collection. (a) The handheld design naturally provides haptic feedback to the demonstrator during contact-rich tasks (e.g., wiping), which is otherwise hard to obtain via teleoperation. Contact microphone captures high-frequency audio feedback that is recorded simultaneously with images. High friction tape is applied on top to augment the signals. (b) shows the domain gap between training and deployment data. (c) shows policy learned from in-the-wild data directly deployed on the robot.
• Figure 3: Network Architecture.
• Figure 4: Attention Visualization. Interestingly, we find that a policy co-trained with audio attends more to the task-relevant regions (shape of drawing or free space inside the pan). In contrast, the vision-only policy often overfits to background structures as a shortcut to estimate contact (e.g., the edge of the whiteboard, table, and room structures).
• Figure 5: Flipping Evaluation. Up: On the left, we show the in-lab test scenarios. We train the policy with in-lab demonstrations collected in the same environment as inference time. We study three types of test-time variations: including different robot and object configurations (T1), audio perturbation when we play different types of noises in the environment (T2), and different table heights than the one in training data (T3). On the right, we show the two unseen environments for the in-the-wild generalization test. Bottom: Typical failure cases and task success rate. The details (e.g., failure cases) for each rollout can be found in the appendix.