Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Using Vision-Language Models as Proxies for Social Intelligence in Human-Robot Interaction

Fanjun Bu, Melina Tsai, Audrey Tjokro, Tapomayukh Bhattacharjee, Jorge Ortiz, Wendy Ju

TL;DR

The paper tackles enabling social intelligence in service robots by leveraging vision-language models (VLMs) as proxies for social reasoning. It introduces a two-stage gated pipeline: Stage 1 detects brief social preambles (gaze shifts and proxemic entries) using lightweight perceptual cues, and Stage 2 uses a video-based VLM to interpret the scene and guide action. Through a five-day Wizard-of-Oz field study in a university cafe, the authors provide an empirical dataset and show that selective VLM invocation can yield plausible, context-aware robot engagement with substantially reduced computational cost. They compare self-consistency and self-critique prompting strategies for uncertainty handling, and discuss design considerations, limitations, and future directions toward open-world social interaction. The approach demonstrates a practical path to endowing robots with adaptable, interpretable social behavior without fully modeling human preferences, emphasizing negotiation and context-driven response in real-world settings.

Abstract

Robots operating in everyday environments must often decide when and whether to engage with people, yet such decisions often hinge on subtle nonverbal cues that unfold over time and are difficult to model explicitly. Drawing on a five-day Wizard-of-Oz deployment of a mobile service robot in a university cafe, we analyze how people signal interaction readiness through nonverbal behaviors and how expert wizards use these cues to guide engagement. Motivated by these observations, we propose a two-stage pipeline in which lightweight perceptual detectors (gaze shifts and proxemics) are used to selectively trigger heavier video-based vision-language model (VLM) queries at socially meaningful moments. We evaluate this pipeline on replayed field interactions and compare two prompting strategies. Our findings suggest that selectively using VLMs as proxies for social reasoning enables socially responsive robot behavior, allowing robots to act appropriately by attending to the cues people naturally provide in real-world interactions.

Using Vision-Language Models as Proxies for Social Intelligence in Human-Robot Interaction

TL;DR

The paper tackles enabling social intelligence in service robots by leveraging vision-language models (VLMs) as proxies for social reasoning. It introduces a two-stage gated pipeline: Stage 1 detects brief social preambles (gaze shifts and proxemic entries) using lightweight perceptual cues, and Stage 2 uses a video-based VLM to interpret the scene and guide action. Through a five-day Wizard-of-Oz field study in a university cafe, the authors provide an empirical dataset and show that selective VLM invocation can yield plausible, context-aware robot engagement with substantially reduced computational cost. They compare self-consistency and self-critique prompting strategies for uncertainty handling, and discuss design considerations, limitations, and future directions toward open-world social interaction. The approach demonstrates a practical path to endowing robots with adaptable, interpretable social behavior without fully modeling human preferences, emphasizing negotiation and context-driven response in real-world settings.

Abstract

Robots operating in everyday environments must often decide when and whether to engage with people, yet such decisions often hinge on subtle nonverbal cues that unfold over time and are difficult to model explicitly. Drawing on a five-day Wizard-of-Oz deployment of a mobile service robot in a university cafe, we analyze how people signal interaction readiness through nonverbal behaviors and how expert wizards use these cues to guide engagement. Motivated by these observations, we propose a two-stage pipeline in which lightweight perceptual detectors (gaze shifts and proxemics) are used to selectively trigger heavier video-based vision-language model (VLM) queries at socially meaningful moments. We evaluate this pipeline on replayed field interactions and compare two prompting strategies. Our findings suggest that selectively using VLMs as proxies for social reasoning enables socially responsive robot behavior, allowing robots to act appropriately by attending to the cues people naturally provide in real-world interactions.

Paper Structure

This paper contains 49 sections, 6 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Using Foundation Models to Achieve Robot Autonomy
  4. Social Cues and Proxemics in HRI
  5. Empirical Data Collection
  6. Robot Platform
  7. Protocol
  8. Dataset
  9. Expert Annotations
  10. Two-Stage Pipeline
  11. Problem Formulation
  12. Two-Stage Gated Decision Process
  13. Stage 1: Preamble Detection Gate
  14. Stage 2: Conditional VLM Inference
  15. Uncertainty Quantification
  16. ...and 34 more sections

Figures (5)

  • Figure 1: Robot Hardware Setup.
  • Figure 2: Velocity signals during a brief glance shift toward and away from the robot (same interaction as Fig. \ref{['fig:teaser']}).
  • Figure 3: Comparing Stage 1 performance with a distance-only baseline.
  • Figure 4: Excerpted VLM outputs from different reasoning strategies for interaction (Figure \ref{['fig:teaser']}).
  • Figure 5: Distance between the robot and the interactant when gaze-shift preambles are observed.