LongNav-R1: Horizon-Adaptive Multi-Turn RL for Long-Horizon VLA Navigation

TL;DR

LongNav-R1 reframes long-horizon embodied navigation as a multi-turn reinforcement learning problem between a Visual-Language-Action policy and its environment. It introduces Horizon-Adaptive Policy Optimization (HAPO), a critic-free, kernel-based baseline that provides dense, horizon-aware advantage estimates without costly value networks. Empirical results on HM3D, MP3D, and OVON demonstrate state-of-the-art performance, strong sample efficiency, and zero-shot real-world generalization, driven by KV-cache rollout, online token pruning, and a warm-start plus online RL training regime. The work offers a scalable pathway for end-to-end VLA navigation with large perceptual-language priors, and the authors commit to open-sourcing the codebase to enable broader adoption and extension.

Abstract

This paper develops LongNav-R1, an end-to-end multi-turn reinforcement learning (RL) framework designed to optimize Visual-Language-Action (VLA) models for long-horizon navigation. Unlike existing single-turn paradigm, LongNav-R1 reformulates the navigation decision process as a continuous multi-turn conversation between the VLA policy and the embodied environment. This multi-turn RL framework offers two distinct advantages: i) it enables the agent to reason about the causal effects of historical interactions and sequential future outcomes; and ii) it allows the model to learn directly from online interactions, fostering diverse trajectory generation and avoiding the behavioral rigidity often imposed by human demonstrations. Furthermore, we introduce Horizon-Adaptive Policy Optimization. This mechanism explicitly accounts for varying horizon lengths during advantage estimation, facilitating accurate temporal credit assignment over extended sequences. Consequently, the agent develops diverse navigation behaviors and resists collapse during long-horizon tasks. Experiments on object navigation benchmarks validate the framework's efficacy: With 4,000 rollout trajectories, LongNav-R1 boosts the Qwen3-VL-2B success rate from 64.3% to 73.0%. These results demonstrate superior sample efficiency and significantly outperform state-of-the-art methods. The model's generalizability and robustness are further validated by its zero-shot performance in long-horizon real-world navigation settings. All source code will be open-sourced upon publication.

Figures (13)

• Figure 1: LongNav-R1 formulates the navigation process as a multi-turn conversation between the VLA policy and the embodied environment. This end-to-end multi-turn RL framework enables the VLA policy to optimize multi-step decision-making based on cumulative, sequential outcomes.
• Figure 2: Comparison of single-turn SFT and multi-turn RL.
• Figure 3: The overall pipeline of LongNav-R1. The framework optimizes the VLA policy through a three-stage iterative process: i) collecting long-horizon trajectories via multi-turn interactive rollout; ii) computing action-level advantages using the proposed horizon-adaptive estimator; and iii) updating the VLA model via the aggregated optimization objective.
• Figure 4: We compare two HAPO variants, using different kernel sizes $\sigma$. Subplot (a) displays the returns in a batch buffer alongside the regressed baselines, while (b) illustrates the value estimation errors.
• Figure 5: RL demonstrates high efficiency, improving performance by 30% with 4k iterations. Evolution of success rate and path efficiency during training. The training pipeline consists of two phases: an initial SFT phase followed by a RL phase.