Learning to Reason and Navigate: Parameter Efficient Action Planning with Large Language Models

TL;DR

This work tackles efficient navigation in REVERIE by introducing PEAP-LLM, a parameter-efficient action planner built around two LLM modules: LGP for extracting goal objects and rooms, and LAP for generating per-location single-step instructions using current visuals. The approach couples these LLM components with a baseline HM3D-DUET policy through a two-stage fine-tuning pipeline—first Supervised Fine-Tuning (SFT) using prefix/LoRA adapters, then Direct Preference Optimization (DPO) using environmental feedback—to suppress hallucinations and biases. Experimental results on the REVERIE benchmark show state-of-the-art performance, with substantial gains in SPL and RGSPL on unseen splits, confirming improved navigation efficiency and remote grounding. The work demonstrates the value of interactive, context-aware language planning in embodied AI, and highlights future directions for reducing computational overhead and enhancing decision-confidence signaling.

Abstract

The remote embodied referring expression (REVERIE) task requires an agent to navigate through complex indoor environments and localize a remote object specified by high-level instructions, such as "bring me a spoon", without pre-exploration. Hence, an efficient navigation plan is essential for the final success. This paper proposes a novel parameter-efficient action planner using large language models (PEAP-LLM) to generate a single-step instruction at each location. The proposed model consists of two modules, LLM goal planner (LGP) and LoRA action planner (LAP). Initially, LGP extracts the goal-oriented plan from REVERIE instructions, including the target object and room. Then, LAP generates a single-step instruction with the goal-oriented plan, high-level instruction, and current visual observation as input. PEAP-LLM enables the embodied agent to interact with LAP as the path planner on the fly. A simple direct application of LLMs hardly achieves good performance. Also, existing hard-prompt-based methods are error-prone in complicated scenarios and need human intervention. To address these issues and prevent the LLM from generating hallucinations and biased information, we propose a novel two-stage method for fine-tuning the LLM, consisting of supervised fine-tuning (STF) and direct preference optimization (DPO). SFT improves the quality of generated instructions, while DPO utilizes environmental feedback. Experimental results show the superiority of our proposed model on REVERIE compared to the previous state-of-the-art.

Figures (8)

• Figure 1: At time step $t_0$, the LLM goal planner specifies the target object and room using the base LLM. Then, the LoRA action planner utilizes the fine-tuned LLM to generate a single-step instruction with the goal-oriented plan, high-level instruction, and visual observation as input. The base LLM is fine-tuned in two SFT and DPO settings using the FGR2R dataset and environmental feedback, respectively.
• Figure 2: Outline of our PEAP-LLM model. At first, we perform the goal-oriented planning task using the LGP module. Then, LAP takes the goal-oriented plan, high-level instruction, and the current visual observation as input to offer a single-step navigation plan. Finally, the policy model predicts the next action and the process is repeated until the episode ends by predicting the stop action or exceeding the maximum number of steps.
• Figure 3: Example of prompting template for LLM goal planner. The red denotes the input parameters.
• Figure 4: Example of prompting template for LoRA action planner. The red denotes the input parameters.
• Figure 5: The process of decomposing the entire R2R instructions into non-overlapping sub-instructions based on the chunk view specifying which sub-instruction corresponds to which location. $x$ to $y$ shows the determined sub-instruction is suitable for going to viewpoint $y$ from viewpoint $x$.