Towards Open-World Mobile Manipulation in Homes: Lessons from the Neurips 2023 HomeRobot Open Vocabulary Mobile Manipulation Challenge

TL;DR

Open Vocabulary Mobile Manipulation (OVMM) challenges robots to find and place unseen objects in unseen homes, combining simulation and real-world evaluation to study perception, planning, and policy learning. The authors implemented a NeurIPS 2023 competition with a modular HomeRobot OVMM benchmark, a reusable HomeRobot library, and containerized evaluation to probe generalization in open environments. Results show a large gap from baseline 0.8% to top performers at 10.8%, with two recurring themes: improved error detection/recovery and stronger integration of perception with decision-making. The study highlights sim-to-real transfer challenges, emphasizes robust perception and error handling, and demonstrates that reproducible, containerized evaluation is essential for progress toward robust, generalizable home-robot manipulation.

Abstract

In order to develop robots that can effectively serve as versatile and capable home assistants, it is crucial for them to reliably perceive and interact with a wide variety of objects across diverse environments. To this end, we proposed Open Vocabulary Mobile Manipulation as a key benchmark task for robotics: finding any object in a novel environment and placing it on any receptacle surface within that environment. We organized a NeurIPS 2023 competition featuring both simulation and real-world components to evaluate solutions to this task. Our baselines on the most challenging version of this task, using real perception in simulation, achieved only an 0.8% success rate; by the end of the competition, the best participants achieved an 10.8\% success rate, a 13x improvement. We observed that the most successful teams employed a variety of methods, yet two common threads emerged among the best solutions: enhancing error detection and recovery, and improving the integration of perception with decision-making processes. In this paper, we detail the results and methodologies used, both in simulation and real-world settings. We discuss the lessons learned and their implications for future research. Additionally, we compare performance in real and simulated environments, emphasizing the necessity for robust generalization to novel settings.

Figures (8)

• Figure 1: We have designed simulated homes with articulated and moveable objects. Policies can be trained and evaluated in simulation (bottom row) with the same control stack as on physical hardware (top row) while performing the OVMM task. Agents that can accurately navigate around, search for, and manipulate objects in simulation were transferred to the real world benchmark. Figure from yenamandra2023homerobot.
• Figure 2: Maps used for real-world testing and evaluation. We ran real-world evaluations on three different environments, testing search, grasping, and placement of different types of objects and given different queries.
• Figure 3: Examples of object manipulation from real-world experiments. The participant teams needed to find and move three objects (fig \ref{['fig:allobjects']}) from the held-out apartment environment: a plastic cup, a toy animal, and a yellow toy lemon.
• Figure 4: Left: challenge Docker files hierarchy. Both the participant and environment docker images are built off of the same image; the environment image actually contains the task itself, and the participant docker image sends actions and receives observations over the network. Right: challenge evaluation infrastructure.
• Figure 5: The architecture of UniTeam's agent. It is build as a state machine with four states: (Find and) Navigate to object, Pick object, (Find and) Navigate to end receptacle, and Place object. In the Find-phases, the Exploration skill is used. The agent's perception model is Detic zhou2022detecting, it segments the object and receptacles in the RGB image. This segmentation, together with other observations such as the pose, joints and a depth image, are the input to the agent.