ReLI: A Language-Agnostic Approach to Human-Robot Interaction

TL;DR

ReLI tackles cross-lingual human-robot instruction grounding by integrating large-scale language and vision foundation models into a language-to-action loop. It grounds multilingual instructions into executable robot actions via four components: multimodal interaction, reasoning-based instruction parsing, visuo-lingual perception, and action execution, with an explicit language-aware confirmation step. The framework demonstrates strong cross-language generalization across 140 languages, achieving about 90% accuracy in instruction parsing and task execution on average, including low-resource and vulnerable languages, in both short- and long-horizon tasks in both simulated and real settings. This work advances inclusive, real-world HRI by enabling natural multilingual interaction and semantic grounding without language-specific retraining.

Abstract

Adapting autonomous agents for real-world industrial, domestic, and other daily tasks is currently gaining momentum. However, in global or cross-lingual application contexts, ensuring effective interaction with the environment and executing unrestricted human-specified tasks regardless of the language remains an unsolved problem. To address this, we propose ReLI, a language-agnostic approach that enables autonomous agents to converse naturally, semantically reason about their environment, and perform downstream tasks, regardless of the task instruction's modality or linguistic origin. First, we ground large-scale pre-trained foundation models and transform them into language-to-action models that can directly provide common-sense reasoning and high-level robot control through natural, free-flow conversational interactions. Further, we perform cross-lingual adaptation of the models to ensure that ReLI generalises across the global languages. To demonstrate ReLI's robustness, we conducted extensive experiments on various short- and long-horizon tasks, including zero- and few-shot spatial navigation, scene information retrieval, and query-oriented tasks. We benchmarked the performance on $140$ languages involving $70K+$ multi-turn conversations. On average, ReLI achieved over $90\%\pm0.2$ accuracy in cross-lingual instruction parsing and task execution success. These results demonstrate its potential to advance natural human-agent interaction in the real world while championing inclusive and linguistic diversity. Demos and resources will be public at: https://linusnep.github.io/ReLI/.

Figures (12)

• Figure 1: Illustration of how ReLI empowers autonomous agents to perform both short- and long-horizon tasks. (a) A natural language instruction $c \in \mathcal{C}_T$ is given regardless of the language $\ell \in \mathcal{L}$ of the task instruction. In (b) and (c), ReLI reasons over the task instruction and autoregressively generates a sequence of action plans, i.e., $Action_{1}, Action_{2}, \dots, Action_{7}$ that accomplishes the given task. (d) It then seeks the user's consent for these action plans (i.e., in the case of multistep actionable commands) before transmitting them to the robot's controller for physical execution. (e) If the user affirms, the parsed instructions will be executed; otherwise, they will be discarded. See Section \ref{['sec3']} for the formal details.
• Figure 2: Overview of ReLI's architecture. For users' commands in languages generalisable by the state-of-the-art LLMs, we decompose ReLI functionality into four main components that involve: (a) language detection and transcription, (b) instruction reasoning, processing and instruction-to-action parsing, (c) knowledge-based visuo-lingual and spatial grounding, and (d) real-world robot control and action execution. See Section \ref{['sec3']} for details.
• Figure 3: ReLI employs a dynamic and event-driven architecture where each user's language input triggers a corresponding response. Additionally, action execution updates are communicated in the same language as the input to ensure seamless bidirectional and linguistically aligned interaction.
• Figure 4: Distributions of the 140 representative languages utilised for ReLI benchmarking. We prioritise the inclusion of low-resource and vulnerable languages in our selection criteria, as we posit that this will rigorously evaluate the robustness and efficacy of our framework (bottom left). Further, to promote inclusive and accessible HRI, we ensured that our selected languages are strategically distributed across the world's continents (top).
• Figure 5: (a) Distribution of task instructions utilised in our benchmarking (see Table \ref{['tab:task_primitives']} for more details). The labels correspond to $G_n$ (zero-shot spatial and goal-directed tasks), $W_c$ (movement commands without location targeting), $Q_i$ (general information and causal queries), $O_n$ (zero- and few-shot object navigation), and $C_r$ (contextual and descriptive reasoning). (b)–(e) show representative tasks in multiple languages, highlighting ReLI’s ability to interpret, plan, and execute diverse natural language commands. See Appendix \ref{['append:b']} for more visual qualitative examples.