Efficient Robotic Policy Learning via Latent Space Backward Planning

TL;DR

This work tackles the challenge of planning for long-horizon robotic tasks under real-time constraints. It introduces Latent Space Backward Planning (LBP), which grounds the final task objective in a latent visual space and recursively predicts intermediate subgoals in a backward, coarse-to-fine manner. By coupling a latent final-goal predictor with a recursive subgoal predictor and a goal-fusion policy, LBP achieves on-task guidance with lower computational cost and reduced error accumulation compared to forward or frame-by-frame planning. Extensive simulations and real-robot experiments demonstrate that LBP outperforms state-of-the-art approaches, particularly on long-horizon multi-stage tasks, and maintains robustness under real-world disturbances. The approach significantly advances real-time, long-horizon robotic control by delivering efficient, accurate planning with adaptable subgoal guidance.

Abstract

Current robotic planning methods often rely on predicting multi-frame images with full pixel details. While this fine-grained approach can serve as a generic world model, it introduces two significant challenges for downstream policy learning: substantial computational costs that hinder real-time deployment, and accumulated inaccuracies that can mislead action extraction. Planning with coarse-grained subgoals partially alleviates efficiency issues. However, their forward planning schemes can still result in off-task predictions due to accumulation errors, leading to misalignment with long-term goals. This raises a critical question: Can robotic planning be both efficient and accurate enough for real-time control in long-horizon, multi-stage tasks? To address this, we propose a Latent Space Backward Planning scheme (LBP), which begins by grounding the task into final latent goals, followed by recursively predicting intermediate subgoals closer to the current state. The grounded final goal enables backward subgoal planning to always remain aware of task completion, facilitating on-task prediction along the entire planning horizon. The subgoal-conditioned policy incorporates a learnable token to summarize the subgoal sequences and determines how each subgoal guides action extraction. Through extensive simulation and real-robot long-horizon experiments, we show that LBP outperforms existing fine-grained and forward planning methods, achieving SOTA performance. Project Page: https://lbp-authors.github.io

Figures (5)

• Figure 1: Illustration of latent space backward planning.
• Figure 2: Overall framework architecture of LBP.
• Figure 3: Left: the entire desktop environment setups of real-world experiments contains a 6 DoF AIRBOT arm and three Logitech C922PRO cameras with different views; Right: (1) Move cups: move both brown cups in front of the white ones; (2) Stack cups: stack all paper cups together; (3) Shift cups: shift all the paper cups to another plate, in a clockwise direction.
• Figure 5: Mean Squared Errors (MSE) between predicted subgoals and corresponding ground truths under forward and backward paradigm.
• Figure 6: Mean Squared Errors (MSE) between predicted subgoals and corresponding ground truths in parallel, forward and backward planning.