Semantic Skill Grounding for Embodied Instruction-Following in Cross-Domain Environments

TL;DR

SemGro tackles cross-domain embodied instruction-following by grounding pretrained semantic skills through an iterative, hierarchical process. It combines an LM-based task planner with a multi-modal skill critic to ground high-level skills into executable low-level actions suitable for the target domain. The approach maps user instructions to a hierarchical skill database, uses a kNN retriever for in-context examples, and demonstrates strong improvements over baselines across 300 cross-domain VirtualHome scenarios, including higher SR, CGC, and Plan scores, as well as better executable-skill identification. This work highlights the practical value of balancing semantic richness with domain-agnostic executability for robust, real-world EIF systems.

Abstract

In embodied instruction-following (EIF), the integration of pretrained language models (LMs) as task planners emerges as a significant branch, where tasks are planned at the skill level by prompting LMs with pretrained skills and user instructions. However, grounding these pretrained skills in different domains remains challenging due to their intricate entanglement with the domain-specific knowledge. To address this challenge, we present a semantic skill grounding (SemGro) framework that leverages the hierarchical nature of semantic skills. SemGro recognizes the broad spectrum of these skills, ranging from short-horizon low-semantic skills that are universally applicable across domains to long-horizon rich-semantic skills that are highly specialized and tailored for particular domains. The framework employs an iterative skill decomposition approach, starting from the higher levels of semantic skill hierarchy and then moving downwards, so as to ground each planned skill to an executable level within the target domain. To do so, we use the reasoning capabilities of LMs for composing and decomposing semantic skills, as well as their multi-modal extension for assessing the skill feasibility in the target domain. Our experiments in the VirtualHome benchmark show the efficacy of SemGro in 300 cross-domain EIF scenarios.

Figures (3)

• Figure 1: Cross-domain EIF. (a) Conventional approaches focus on single-domain deployment, where the pretrained skills are effective within only that particular domain (marked in the green boxes). (b) Our $\textnormal{SemGro}$ caters to cross-domain deployment, where the pretrained skills may be inexecutable (marked by red boxes) in the target environment. For this, $\textnormal{SemGro}$ explores the semantic hierarchy of skills.
• Figure 2: Overview of $\textnormal{SemGro}$. Given a user instruction $i$ and detected object names from visual observations $o_t$, $\textnormal{SemGro}$ selects $k$ entries (marked in blue dotted entries) from the hierarchical skill database $\mathcal{D}$. Within $\mathcal{D}$, skills are labeled based on their executability in the target domain: non-executable skills in red boxes and executable ones in green. From these selections, in-context examples $\mathbf{x}(i, o_t)$ and skill candidates $\mathbf{c}(i, o_t)$ are obtained from the constituent low-level skills (marked in red dotted entries in $\mathcal{D}$) of these entries. The skill generator $\phi_G$ is then prompted with $\mathbf{x}(i, o_t)$, $\mathbf{c}(i, o_t)$, $i$, and skill execution history $h$ to generate the skill $\bar{l} \in \mathbf{c}(i, o_t)$. The LM-part $\psi_{\text{LM}}$ of the critic $\psi$ assesses the feasibility of the planned skill $\bar{l}$ based on the detected objects $dn(o_t)$ and their states $ds(o_t)$ as identified by the VLM-part $\psi_{\text{VLM}}$. If $\bar{l}$ is deemed executable (Case 1), the agent proceeds to interact with the environment by performing the skill. Otherwise (Case 2), the task retriever $\phi_R$ generates a more fine-grained instruction $i^*$, derived from lower-level skills $\mathbf{lc}(i, o_t)$ (marked in purple dotted entries in $\mathcal{D}$) and feedback $f$ explaining the inexecutability. The $i^*$ is treated as a new instruction, prompting the framework to iterate the process.
• Figure 3: Caption