Open-Vocabulary Mobile Manipulation Based on Double Relaxed Contrastive Learning with Dense Labeling

TL;DR

The paper tackles open-vocabulary fetch-and-carry for domestic service robots by retrieving target objects and receptacles from pre-collected indoor images and executing a carry task. It introduces RelaX-Former, a four-module architecture (SOG, XF, DRL, and OVP encoder) with a double relaxed contrastive loss and a Dense Labeler to better handle unlabeled positives. On the LTRRIE-FC dataset, RelaX-Former outperforms strong baselines in image retrieval, and in real-world zero-shot experiments it achieves a 75% success rate, demonstrating practical viability. The approach integrates segmentation-guided, multimodal fused representations with open-vocabulary language processing to enable robust, open-ended manipulation in unseen environments.

Abstract

Growing labor shortages are increasing the demand for domestic service robots (DSRs) to assist in various settings. In this study, we develop a DSR that transports everyday objects to specified pieces of furniture based on open-vocabulary instructions. Our approach focuses on retrieving images of target objects and receptacles from pre-collected images of indoor environments. For example, given an instruction "Please get the right red towel hanging on the metal towel rack and put it in the white washing machine on the left," the DSR is expected to carry the red towel to the washing machine based on the retrieved images. This is challenging because the correct images should be retrieved from thousands of collected images, which may include many images of similar towels and appliances. To address this, we propose RelaX-Former, which learns diverse and robust representations from among positive, unlabeled positive, and negative samples. We evaluated RelaX-Former on a dataset containing real-world indoor images and human annotated instructions including complex referring expressions. The experimental results demonstrate that RelaX-Former outperformed existing baseline models across standard image retrieval metrics. Moreover, we performed physical experiments using a DSR to evaluate the performance of our approach in a zero-shot transfer setting. The experiments involved the DSR to carry objects to specific receptacles based on open-vocabulary instructions, achieving an overall success rate of 75%.

Figures (4)

• Figure 1: Overview of our task. First, the DSR collects images of the environment through pre-exploration. Given an open-vocabulary instruction, it is required to retrieve the red and blue framed images as the target object image and receptacle image, respectively, from the collected images. Subsequently, the DSR carries the target object to the receptacle, based on the user-selected images.
• Figure 2: Architecture of RelaX-Former. The proposed architecture consists of four modules: Spatial Overlay Grounding (SOG) module, X-Fusion (XF) module, Dense Representation Learning (DRL) module, and Open-Vocabulary Phrase (OVP) encoder. Here, $N$ denotes the batch size.
• Figure 3: Qualitative results of RelaX-Former and the most competitive baseline method korekata24arxiv from the HM3D-FC test for $\bm{x}_{\text{txt}}$. The ground-truth image and the top-3 retrieved images are shown for each mode. (a) Target mode and (b) receptacle mode. Positive, unlabeled positive, and negative labels are colored in green, yellow, and red, respectively.
• Figure 4: Qualitative results of the physical experiments with a given instruction $\bm{x}_{\text{txt}}$. The top-3 retrieved images for each mode are shown, alongside the scenes of fetching and carrying actions. The images that were considered correct or incorrect for each mode are framed in green or red, respectively.