Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Survey of Language-Based Communication in Robotics

William Hunt, Sarvapali D. Ramchurn, Mohammad D. Soorati

TL;DR

The paper surveys language-based robotics, addressing how natural language interfaces and large language models can augment robot planning, control, perception, and interaction. It provides a taxonomy by human–robot–robot communication pathways and internal architectures, detailing methods for task planning, code and reward generation, explainability, and dynamic adaptation. It highlights applications in autonomous systems, human–robot collaboration, and ad-hoc multi-robot coordination, while candidly discussing hallucination, security, memory, data, and safety challenges. The work emphasizes that while language can substantially enhance flexibility and collaboration, practical deployments will rely on hybrid architectures that combine high-level language reasoning with reliable, domain-specific controllers and governance frameworks to ensure safety and trustworthiness.

Abstract

Embodied robots which can interact with their environment and neighbours are increasingly being used as a test case to develop Artificial Intelligence. This creates a need for multimodal robot controllers that can operate across different types of information, including text. Large Language Models are able to process and generate textual as well as audiovisual data and, more recently, robot actions. Language Models are increasingly being applied to robotic systems; these Language-Based robots leverage the power of language models in a variety of ways. Additionally, the use of language opens up multiple forms of information exchange between members of a human-robot team. This survey motivates the use of language models in robotics, and then delineates works based on the part of the overall control flow in which language is incorporated. Language can be used by human to task a robot, by a robot to inform a human, between robots as a human-like communication medium, and internally for a robot's planning and control. Applications of language-based robots are explored, and numerous limitations and challenges are discussed to provide a summary of the development needed for the future of language-based robotics.

A Survey of Language-Based Communication in Robotics

TL;DR

The paper surveys language-based robotics, addressing how natural language interfaces and large language models can augment robot planning, control, perception, and interaction. It provides a taxonomy by human–robot–robot communication pathways and internal architectures, detailing methods for task planning, code and reward generation, explainability, and dynamic adaptation. It highlights applications in autonomous systems, human–robot collaboration, and ad-hoc multi-robot coordination, while candidly discussing hallucination, security, memory, data, and safety challenges. The work emphasizes that while language can substantially enhance flexibility and collaboration, practical deployments will rely on hybrid architectures that combine high-level language reasoning with reliable, domain-specific controllers and governance frameworks to ensure safety and trustworthiness.

Abstract

Embodied robots which can interact with their environment and neighbours are increasingly being used as a test case to develop Artificial Intelligence. This creates a need for multimodal robot controllers that can operate across different types of information, including text. Large Language Models are able to process and generate textual as well as audiovisual data and, more recently, robot actions. Language Models are increasingly being applied to robotic systems; these Language-Based robots leverage the power of language models in a variety of ways. Additionally, the use of language opens up multiple forms of information exchange between members of a human-robot team. This survey motivates the use of language models in robotics, and then delineates works based on the part of the overall control flow in which language is incorporated. Language can be used by human to task a robot, by a robot to inform a human, between robots as a human-like communication medium, and internally for a robot's planning and control. Applications of language-based robots are explored, and numerous limitations and challenges are discussed to provide a summary of the development needed for the future of language-based robotics.
Paper Structure (42 sections, 1 figure, 4 tables)

This paper contains 42 sections, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Why Use Language?
  3. Knowledge and Inference
  4. Zero-shot Adaptability
  5. Interactable and Understandable
  6. Biological Inspiration and Language-as-Intelligence
  7. The Paradigm for Communication in Multi-Robot Systems
  8. Human to Robot Communication
  9. Task Breakdown and Planning
  10. Writing Code and Defining Rewards
  11. Direct Integration of Human Input
  12. Robot to Human Communication
  13. Explainability
  14. Asking for Human Assistance
  15. Robot to Robot Communication
  16. ...and 27 more sections

Figures (1)

  • Figure 1: Overview of this survey. Works are categorised by direction of communication between members of the human-robot team. Advantages, applications, and limitations are also discussed.