SocialNav-MoE: A Mixture-of-Experts Vision Language Model for Socially Compliant Navigation with Reinforcement Fine-Tuning

TL;DR

SocialNav-MoE introduces an efficient mixture-of-experts vision-language framework for socially compliant robot navigation, addressing the computational burden of large VLMs with reinforcement fine-tuning and a semantic similarity reward. The method systematically evaluates small backbones (Phi, Qwen, StableLM), routing strategies, and vision encoders (CLIP, SigLIP), showing that four experts with top-1 routing, a frozen SigLIP encoder, and SSR-based RFT achieve strong social- navigation performance with high inference speed on the SNEI dataset. A novel SSR reward bridges sparse rewards and semantic alignment, improving in-loop decision-making for social contexts. The work provides practical design guidelines and demonstrates competitive efficiency and semantic alignment compared to large generalist baselines, suggesting viability for real-time, resource-constrained robotic deployments.

Abstract

For robots navigating in human-populated environments, safety and social compliance are equally critical, yet prior work has mostly emphasized safety. Socially compliant navigation that accounts for human comfort, social norms, and contextual appropriateness remains underexplored. Vision language models (VLMs) show promise for this task; however, large-scale models incur substantial computational overhead, leading to higher inference latency and energy consumption, which makes them unsuitable for real-time deployment on resource-constrained robotic platforms. To address this issue, we investigate the effectiveness of small VLM and propose SocialNav-MoE, an efficient Mixture-of-Experts vision language model for socially compliant navigation with reinforcement fine-tuning (RFT). We further introduce a semantic similarity reward (SSR) to effectively leverage RFT for enhancing the decision-making capabilities. Additionally, we study the effectiveness of different small language model types (Phi, Qwen, and StableLM), routing strategies, and vision encoders (CLIP vs. SigLIP, frozen vs. fine-tuned). Experiments on the SNEI dataset demonstrate that SocialNav-MoE achieves an excellent balance between navigation accuracy and efficiency. The proposed SSR function is more effective than hard-level and character-level rewards. Source code will be released upon acceptance.

Figures (5)

• Figure 1: The overall architecture of SocialNav-MoE. SocialNav-MoE consists of three training stages: supervised fine-tuning, reinforcement fine-tuning, and MoE fine-tuning.
• Figure 2: SNEI dataset example.
• Figure 3: Visualization of the model outputs. The blue arrow indicates the ground truth, the red arrow denotes our method, the yellow arrow represents GPT-4o, and the green arrow corresponds to Claude. GPT-4o and Claude often fail to produce socially compliant motions. For example, in the top-right image, GPT-4o predicts a forward action and Claude predicts a stop action, where a slight right turn would be more appropriate. In the central top and bottom images, GPT-4o also suggests moving forward when stopping is preferable. These cases highlight the advantages of our approach in maintaining fluid and socially aware navigation.
• Figure 4: Visualization of action outputs with and without RFT. Arrows indicate trajectories: blue for ground truth (GT), red for SocialNav-MoE with RFT, and yellow for SocialNav-MoE without RFT. Left: With RFT, the model correctly aligns with the GT by predicting a slight right turn, whereas without RFT it fails to deviate and proceeds forward at a moderate speed. Right: Although both models predict a straight trajectory, the RFT-enhanced model accurately matches the GT’s low speed, while the model without RFT incorrectly predicts a higher speed. These examples demonstrate that RFT substantially improves socially compliant navigation.
• Figure 5: Visualization of failure cases. The blue arrow indicates the ground truth (GT), the red arrow denotes our method. Left: GT suggests continuing straight at a slow speed, whereas our model predicts a slight right turn; Right: GT favors a slight right turn, while our model continues forward at a moderate speed. Both decisions are socially acceptable, reflecting the inherent ambiguity in defining universally valid navigation norms. These cases illustrate that deviations from GT do not always correspond to clear mistakes, but rather to alternative interpretations of socially compliant behavior.