Probing Prompt Design for Socially Compliant Robot Navigation with Vision Language Models

TL;DR

The paper investigates principled prompt design to enhance socially compliant navigation using small vision-language models. It introduces two coupled design axes—system guidance and motivational framing—and constructs a catalog of prompts evaluated on non-finetuned GPT-4o and finetuned TinyLLaVA-based VLMs across two datasets. Results reveal that zero-shot GPT-4o benefits from reasoning prompts with human competition, while finetuned VLMs benefit from perception–reasoning prompts with past-self competition; importantly, poorly designed prompts can degrade performance and prompt design largely constrains decision-making rather than semantic understanding. The findings offer practical guidance for deploying efficient small VLMs in real-world robot navigation, demonstrating substantial gains in action accuracy without enlarging model size.

Abstract

Language models are increasingly used for social robot navigation, yet existing benchmarks largely overlook principled prompt design for socially compliant behavior. This limitation is particularly relevant in practice, as many systems rely on small vision language models (VLMs) for efficiency. Compared to large language models, small VLMs exhibit weaker decision-making capabilities, making effective prompt design critical for accurate navigation. Inspired by cognitive theories of human learning and motivation, we study prompt design along two dimensions: system guidance (action-focused, reasoning-oriented, and perception-reasoning prompts) and motivational framing, where models compete against humans, other AI systems, or their past selves. Experiments on two socially compliant navigation datasets reveal three key findings. First, for non-finetuned GPT-4o, competition against humans achieves the best performance, while competition against other AI systems performs worst. For finetuned models, competition against the model's past self yields the strongest results, followed by competition against humans, with performance further influenced by coupling effects among prompt design, model choice, and dataset characteristics. Second, inappropriate system prompt design can significantly degrade performance, even compared to direct finetuning. Third, while direct finetuning substantially improves semantic-level metrics such as perception, prediction, and reasoning, it yields limited gains in action accuracy. In contrast, our system prompts produce a disproportionately larger improvement in action accuracy, indicating that the proposed prompt design primarily acts as a decision-level constraint rather than a representational enhancement.

Figures (4)

• Figure 1: Non-finetuned GPT-4o performs best with reasoning-focused prompts and human-competition framing, while finetuned VLMs benefit most from perception–reasoning prompts combined with competition against the model’s past behavior.
• Figure 2: Overview of the TinyLLaVA framework used in our experiments.
• Figure 3: Visualizations of the samples from SNEI and MUSON datasets.
• Figure 4: Visualization of perception, prediction, reasoning, and decision making for the finetuned model using TinyLlama-1.1B-Chat-v1.0 as the language model. The results indicate that prompts integrating perception and reasoning with competition against the model’s past behavior achieve the best overall performance.