Unifying Large Language Model and Deep Reinforcement Learning for Human-in-Loop Interactive Socially-aware Navigation

TL;DR

This paper tackles socially aware navigation in human-filled spaces under real-time user inputs and zero-shot environments. It proposes SALM, a unified framework that combines a Large Language Navigation Model (LNM), a DRL-based RLNM, and a Large Language Feedback Model (LFM) with a memory mechanism and Graph-of-Thought reasoning to adapt actions through user language and feedback. The approach is formalized as a Decentralized Partially Observable Semi-Markov Decision Process (Dec-POSMDP) and demonstrated in simulation where SALM outperforms traditional planners and ablations on both success rate and social score, highlighting improved adaptability, personalization, and robustness. The work advances interactive, human-in-the-loop social navigation by leveraging LLMs for high-level guidance and DRL for reliable low-level control, with GoT-based evaluation enabling dynamic fusion of complementary planning strategies.

Abstract

Navigating human-filled spaces is crucial for the interactive social robots to support advanced services, such as cooperative carrying, which enables service provision in complex and crowded environments while adapting behavior based on real-time human language commands or feedback. However, existing social robot navigation planners face two major challenges: managing real-time user inputs and ensuring socially compliant behaviors in unfamiliar, zero-shot environments. In response, we introduce SALM, an interactive, human-in-loop Socially-Aware navigation Large Language Model framework that dynamically integrates deep reinforcement learning (DRL) with large language model (LLM) capabilities. SALM leverages contextual semantic understanding from real-time human-robot interactions to convert high-level user commands into precise, low-level control actions. A high-level LLM module parses user input, guiding the simultaneous generation of navigation commands by both a large language navigation model (LNM) and a DRL-based navigation model (RLNM). A memory mechanism archives temporal data for continuous refinement, while a multi-step graph-of-thoughts inference-based large language feedback model adaptively fuses the strengths of both planning approaches. Experimental evaluations demonstrate that SALM not only enhances navigational precision in crowded, dynamic environments but also significantly improves system adaptability, offering tailored behaviors that align with individual user preferences and real-time feedback. More details and videos about this work are available at: https://sites.google.com/view/navi-salm.

Figures (5)

• Figure 1: An illustration of human-in-the-loop interactive social navigation: integrating DRL and LLM enables the social robot to adaptively respond to environmental changes such as task target and preferred social distance from real-time human feedback.
• Figure 2: SALM architecture: SALM is implemented as a human-in-loop interactive social robot navigation framework, which executes human commands based on LM-based planner, feedback-based planner, and DRL-based planner incorporating. Firstly, users' requests or real-time feedback are processed or replanned to high-level task guidance for three action executors via LLM. Then, the image-to-text encoder and spatio-temporal graph HRI encoder convert robot local observation information to features as LNM and RLNM input, which generate RL-based action, LM-based action, and feedback-based action. Lastly, the above three actions are adaptively fused by a low-level execution decoder as the robot behavior output of SALM.
• Figure 3: An illustration of large language navigation model (LNM): The prompt engineering of LNM comprises task description, global guidance, data annotation, initialization, historical data, additional information, and encoded state to directly generate low-level robot actions $[v_x, v_y]$.
• Figure 4: LFM framework: LFM reconciles the output from LNM $\mathrm{\mathbf{a}^{LM}}$ and RLNM $\mathrm{\mathbf{a}^{RL}}$ to stabilize final mixture action $\mathrm{\mathbf{a}^{R}}$, in which the GoT construction of LFM is designed to evaluate and score the above two executions with more generated evidences or intermediate steps chains from different perspectives.
• Figure 5: The illustration of human-in-loop interactive social navigation: The social robot is navigating toward the red star destination with a blue circle user across ten green circle humans. User's feedback is randomly generated with $50\%$ probability to robot(e.g. change the goal position).