Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Social-LLaVA: Enhancing Robot Navigation through Human-Language Reasoning in Social Spaces

Amirreza Payandeh, Daeun Song, Mohammad Nazeri, Jing Liang, Praneel Mukherjee, Amir Hossain Raj, Yangzhe Kong, Dinesh Manocha, Xuesu Xiao

TL;DR

The paper tackles socially compliant robot navigation by bridging perception to action through language-based reasoning using Vision-Language Models. It introduces the snei dataset (over 40K human-annotated VQA across 2K social scenarios) and a Social-LLaVA model fine-tuned on snei to produce chain-of-thought style language explanations and high-level navigation guidance. Evaluations include qualitative human judgments showing Social-LLaVA outperforming GPT-4V and Gemini on social navigation VQA and a real-world onboard demonstration, highlighting improved explainability and decision-making. These contributions point toward more intuitive, language-grounded, and socially aware robot navigation in unstructured public spaces, while acknowledging limitations and proposing directions for richer data and grounding to action.

Abstract

Most existing social robot navigation techniques either leverage hand-crafted rules or human demonstrations to connect robot perception to socially compliant actions. However, there remains a significant gap in effectively translating perception into socially compliant actions, much like how human reasoning naturally occurs in dynamic environments. Considering the recent success of Vision-Language Models (VLMs), we propose using language to bridge the gap in human-like reasoning between perception and socially aware robot actions. We create a vision-language dataset, Social robot Navigation via Explainable Interactions (SNEI), featuring 40K human-annotated Visual Question Answers (VQAs) based on 2K human-robot social interactions in unstructured, crowded public spaces, spanning perception, prediction, chain-of-thought reasoning, action, and explanation. We fine-tune a VLM, Social-LLaVA, using SNEI to demonstrate the practical application of our dataset. Social-LLaVA outperforms state-of-the-art models like GPT-4V and Gemini, based on the average of fifteen different human-judge scores across 50 VQA. Deployed onboard a mobile robot, Social-LLaVA enables human-like reasoning, marking a promising step toward socially compliant robot navigation in dynamic public spaces through language reasoning.

Social-LLaVA: Enhancing Robot Navigation through Human-Language Reasoning in Social Spaces

TL;DR

The paper tackles socially compliant robot navigation by bridging perception to action through language-based reasoning using Vision-Language Models. It introduces the snei dataset (over 40K human-annotated VQA across 2K social scenarios) and a Social-LLaVA model fine-tuned on snei to produce chain-of-thought style language explanations and high-level navigation guidance. Evaluations include qualitative human judgments showing Social-LLaVA outperforming GPT-4V and Gemini on social navigation VQA and a real-world onboard demonstration, highlighting improved explainability and decision-making. These contributions point toward more intuitive, language-grounded, and socially aware robot navigation in unstructured public spaces, while acknowledging limitations and proposing directions for richer data and grounding to action.

Abstract

Most existing social robot navigation techniques either leverage hand-crafted rules or human demonstrations to connect robot perception to socially compliant actions. However, there remains a significant gap in effectively translating perception into socially compliant actions, much like how human reasoning naturally occurs in dynamic environments. Considering the recent success of Vision-Language Models (VLMs), we propose using language to bridge the gap in human-like reasoning between perception and socially aware robot actions. We create a vision-language dataset, Social robot Navigation via Explainable Interactions (SNEI), featuring 40K human-annotated Visual Question Answers (VQAs) based on 2K human-robot social interactions in unstructured, crowded public spaces, spanning perception, prediction, chain-of-thought reasoning, action, and explanation. We fine-tune a VLM, Social-LLaVA, using SNEI to demonstrate the practical application of our dataset. Social-LLaVA outperforms state-of-the-art models like GPT-4V and Gemini, based on the average of fifteen different human-judge scores across 50 VQA. Deployed onboard a mobile robot, Social-LLaVA enables human-like reasoning, marking a promising step toward socially compliant robot navigation in dynamic public spaces through language reasoning.
Paper Structure (20 sections, 3 figures, 1 table)

This paper contains 20 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Social Navigation
  4. Visual Instruction Tuning
  5. The snei Dataset
  6. Motivation
  7. Data Construction and Analyses
  8. Categorical Labels
  9. Free-form natural language
  10. snei Applications
  11. Social-LLaVA
  12. Qualitative VQA Evaluation
  13. Real World Robot Experiment
  14. Discussions
  15. Limited Spatial Understanding
  16. ...and 5 more sections

Figures (3)

  • Figure 1: Bridging perception to socially compliant action through Chain-of-Thought reasoning using human language.
  • Figure 2: Social-LLaVA: Proof-of-concept real-world experiment demonstrating the robot's ability to understand context and social cues to navigate, thereby avoiding interrupting people's conversations.
  • Figure 3: Qualitative results of our Social-LLaVA model fine-tuned on our snei dataset compared against GPT4-V openai2024gpt4technicalreport and Gemini 1.5 Pro team2023gemini. (a) shows the visual input given to the models. Note that the given scenario is where a robot navigates through a narrow passage partially obstructed by a pillar (colored red), while an individual approaches the robot (red arrow). (b) illustrates the output from Social-LLaVA, while (c) provides comparisons with GPT4-V and Gemini 1.5 Pro. Phrases in blue indicate accurate reasoning and socially compliant results, while highlighted phrases mark instances of hallucination.