QuASH: Using Natural-Language Heuristics to Query Visual-Language Robotic Maps

TL;DR

QuASH tackles open-vocabulary querying in robotic semantic maps by augmenting user queries with semantic synonyms and antonyms and training a non-linear classifier in the shared latent space to estimate the region corresponding to a query. This encoder-agnostic approach replaces simple cosine-thresholding, enabling more accurate, non-isotropic region estimation across both image benchmarks and latent-semantic maps. Extensive ablations show that leveraging language-distribution knowledge improves precision and overall F1-score, especially with OpenSeg, and highlight the importance of thoughtful synonym/antonym selection. The work advances practical open-vocabulary semantics for robotic perception, with strong implications for adaptable scene understanding and retrieval across modalities.

Abstract

Embeddings from Visual-Language Models are increasingly utilized to represent semantics in robotic maps, offering an open-vocabulary scene understanding that surpasses traditional, limited labels. Embeddings enable on-demand querying by comparing embedded user text prompts to map embeddings via a similarity metric. The key challenge in performing the task indicated in a query is that the robot must determine the parts of the environment relevant to the query. This paper proposes a solution to this challenge. We leverage natural-language synonyms and antonyms associated with the query within the embedding space, applying heuristics to estimate the language space relevant to the query, and use that to train a classifier to partition the environment into matches and non-matches. We evaluate our method through extensive experiments, querying both maps and standard image benchmarks. The results demonstrate increased queryability of maps and images. Our querying technique is agnostic to the representation and encoder used, and requires limited training.

Figures (5)

• Figure 1: Our method, QuASH, works by augmenting a single query with a semantic heuristic to acquire a set of semantic synonyms and antonyms to train a classifier in the latent space of a . The classifier enables non-linear classification in the latent space, which in turn enables accurate estimation of the region in an image or a map that corresponds to the query.
• Figure 2: The state-of-the-art solutions implicitly classify the embedding space linearly by thresholding cosine similarities (a) jatavallabhula_conceptfusion_2023, or by using a single complementary embedding (b) huang_visual_2023. By using a with a non-linear kernel, our proposed method (c) can classify the embeddings non-linearly. Embeddings matching the query are visualized in red, and those not matching are visualized in blue. A diamond represents the query and its complement, and dashed lines represent the thresholds.
• Figure 3: The intuition behind our method is that, as the textual descriptions and visual features are embedded into the same space $\mathcal{E}$, training a classifier on the embeddings of the text descriptions will result in a good classifier in the visual domain. Instead of using just a single example, i.e., the query $q$, represented as a black dot, we additionally sample semantically close words, represented as circles, to form a set $\mathcal{Q}$, as well as a non-matching queries $\bar{q} \in \bar{\mathcal{Q}}$, represented as crosses, from the language space $\mathcal{L}$. Using these, we aim to capture the extent of the region $\mathcal{R_{\mathcal{Q}}}$ in the embedding space $\mathcal{E}$. Knowing $\mathcal{R_{\mathcal{Q}}}$ enables finding the visual region $\mathcal{V}_{\mathcal{Q}}$ corresponding to the query $q$.
• Figure 4: Examples of predictions from the COCO data set. The original image is shown on the top row, our method using an SVM classifier is on the second row, and the baseline is on the third row. Our method is generally slightly more accurate in contouring objects, as shown in the first column, and can distinguish related objects better, shown in the second column, and is less prone to overprediction, as shown in the third and fourth columns.
• Figure 5: Examples of map query experiment results. Our method with "ground truth" synonyms is shown in the top row, and the baseline is shown in the bottom row, with the same query displayed in each column. With LSeg (a), (e), the results are very similar. On the other hand, with OpenSeg, we can clearly see the difference, particularly in the significantly improved precision of our method. However, the drawback is that our method suffers from reduced recall, as evident in the failure cases in (d) and (h). The predictions are shown in green, and the outline of the ground truth in red.