Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Instruction to Event: Sound-Triggered Mobile Manipulation

Hao Ju, Shaofei Huang, Hongyu Li, Zihan Ding, Si Liu, Meng Wang, Zhedong Zheng

TL;DR

Sound-triggered mobile manipulation replaces predefined textual instructions with acoustic event cues, enabling robots to autonomously perceive and respond to sound-emitting objects. The authors introduce Habitat-Echo, a simulation platform that renders realistic audio via Room Impulse Responses and couples it with physical interaction, and propose a hierarchical baseline with an Omni-LLM task planner and specialized policy models. They formalize three tasks—SonicStow, SonicInteract, and Bi-Sonic Manipulation—covering single and dual-source sound scenarios, and demonstrate robust performance, including isolating the primary source amidst interference in Bi-Sonic. The work advances embodied AI by enabling timely, audio-driven manipulation in dynamic environments, with potential implications for home robotics and assistive automation.

Abstract

Current mobile manipulation research predominantly follows an instruction-driven paradigm, where agents rely on predefined textual commands to execute tasks. However, this setting confines agents to a passive role, limiting their autonomy and ability to react to dynamic environmental events. To address these limitations, we introduce sound-triggered mobile manipulation, where agents must actively perceive and interact with sound-emitting objects without explicit action instructions. To support these tasks, we develop Habitat-Echo, a data platform that integrates acoustic rendering with physical interaction. We further propose a baseline comprising a high-level task planner and low-level policy models to complete these tasks. Extensive experiments show that the proposed baseline empowers agents to actively detect and respond to auditory events, eliminating the need for case-by-case instructions. Notably, in the challenging dual-source scenario, the agent successfully isolates the primary source from overlapping acoustic interference to execute the first interaction, and subsequently proceeds to manipulate the secondary object, verifying the robustness of the baseline.

From Instruction to Event: Sound-Triggered Mobile Manipulation

TL;DR

Sound-triggered mobile manipulation replaces predefined textual instructions with acoustic event cues, enabling robots to autonomously perceive and respond to sound-emitting objects. The authors introduce Habitat-Echo, a simulation platform that renders realistic audio via Room Impulse Responses and couples it with physical interaction, and propose a hierarchical baseline with an Omni-LLM task planner and specialized policy models. They formalize three tasks—SonicStow, SonicInteract, and Bi-Sonic Manipulation—covering single and dual-source sound scenarios, and demonstrate robust performance, including isolating the primary source amidst interference in Bi-Sonic. The work advances embodied AI by enabling timely, audio-driven manipulation in dynamic environments, with potential implications for home robotics and assistive automation.

Abstract

Current mobile manipulation research predominantly follows an instruction-driven paradigm, where agents rely on predefined textual commands to execute tasks. However, this setting confines agents to a passive role, limiting their autonomy and ability to react to dynamic environmental events. To address these limitations, we introduce sound-triggered mobile manipulation, where agents must actively perceive and interact with sound-emitting objects without explicit action instructions. To support these tasks, we develop Habitat-Echo, a data platform that integrates acoustic rendering with physical interaction. We further propose a baseline comprising a high-level task planner and low-level policy models to complete these tasks. Extensive experiments show that the proposed baseline empowers agents to actively detect and respond to auditory events, eliminating the need for case-by-case instructions. Notably, in the challenging dual-source scenario, the agent successfully isolates the primary source from overlapping acoustic interference to execute the first interaction, and subsequently proceeds to manipulate the secondary object, verifying the robustness of the baseline.
Paper Structure (18 sections, 5 figures, 3 tables)

This paper contains 18 sections, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Task Definition and Skill
  4. Task Definition
  5. Skill
  6. Simulator and Benchmark
  7. Simulator: Habitat-Echo
  8. Benchmark
  9. Baseline
  10. Sound-triggered Task Planner
  11. Policy Model
  12. Experiment
  13. Implementation
  14. Result
  15. Ablation Study and Further Discussion
  16. ...and 3 more sections

Figures (5)

  • Figure 1: Motivation for our work. (Upper) An example of sound-triggered mobile manipulation, the trigger signal is the sound of the event rather than instructions from humans. (Left) Instruction-based mobile manipulation needs humans to analyze the trigger signal and manually give the instruction for the robot model, which is passive. (Right) In sound-triggered mobile manipulation, the robot model automatically receives sound signals and actively outputs executable actions.
  • Figure 2: Illustration of the proposed tasks. (a) SonicStow requires the agent to interact with the sound source (a rigid object) via Navigate, Pick, and Place skills. (b) SonicInteract requires the agent to interact with the sound source (an articulated object) via Navigate, Open Door, and Close Sink skills. (c) Bi-Sonic Manipulation requires the agent to interact with the sound source (two objects sequentially) via Navigate, Pick, Place, Open Door, and Close Sink skills.
  • Figure 3: Overview of the proposed baseline. The sound-triggered task planner processes initial audio-visual observations to reason and generate a high-level skill chain from the skill library. Guided by this chain, specialized policy models are sequentially activated to generate low-level actions and interact with the environment.
  • Figure 4: Qualitative Visualization of Task Execution. We present the execution trajectories for (a) SonicStow, (b) SonicInteract, and (c) Bi-Sonic Manipulation. For each task, the left panel illustrates the top-down view of the environment, highlighting the agent's trajectories and the location of sound sources. The right panel displays the corresponding sequence of third-person keyframes during the execution. Note that Bi-Sonic Manipulation involves navigating to and interacting with two distinct sound sources sequentially.
  • Figure 5: Additional Qualitative Results of Different Skills. (a), (b) Success and failure cases of the Pick skill. (c), (d) Success and failure cases of the Place skill. (e), (f) Success and failure cases of the Open Door skill. (g), (h) Success and failure cases of the Close Sink skill.