Visual Foresight for Robotic Stow: A Diffusion-Based World Model from Sparse Snapshots

TL;DR

The paper tackles visual foresight for robotic bin stow in real warehouses, where only sparse pre- and post-stow snapshots are available. It introduces FOREST, a stow-intent-conditioned latent diffusion world model that operates on slot-aligned item masks and is conditioned on the pre-stow state, the new item, and the planned stow intent. Through a three-stage pipeline—signal extraction via instance masks and Hungarian matching, token-based input modeling, and a transformer-based latent diffusion architecture—FOREST achieves strong direct IoU improvements over heuristic baselines and provides useful foresight signals for downstream tasks like DLO prediction and multi-stow reasoning. Experiments on ARMBench demonstrate substantial gains in post-stow layout accuracy and reveal that forecasting post-stow layouts with FOREST leads to only modest degradations in downstream metrics, highlighting the practical potential of learned world models for warehouse planning and policy evaluation. $ ext{FOREST}: ext{ a stow-intent-conditioned diffusion-based world model that maps } (x_{pre}, o_{new}, u) ext{ to } ilde{x}_{post} ext{ with } ilde{x}_{post} ext{ represented as slot-aligned masks.}$

Abstract

Automated warehouses execute millions of stow operations, where robots place objects into storage bins. For these systems it is valuable to anticipate how a bin will look from the current observations and the planned stow behavior before real execution. We propose FOREST, a stow-intent-conditioned world model that represents bin states as item-aligned instance masks and uses a latent diffusion transformer to predict the post-stow configuration from the observed context. Our evaluation shows that FOREST substantially improves the geometric agreement between predicted and true post-stow layouts compared with heuristic baselines. We further evaluate the predicted post-stow layouts in two downstream tasks, in which replacing the real post-stow masks with FOREST predictions causes only modest performance loss in load-quality assessment and multi-stow reasoning, indicating that our model can provide useful foresight signals for warehouse planning.

Figures (16)

• Figure 1: Examples of visual foresight for robotic stow.
• Figure 2: Examples in ARMBench with pre-stow RGB, post-stow RGB, and induct-view new item image. The stow intents are depicted by bounding box overlayed on the pre-stow RGB image, in which green one refers to the planned position for new item placement while the blue one is the location of sweeping operation.
• Figure 3: Overview of FOREST, our diffusion-based world model for visual foresight of stow with three stages. Stage 1 constructs slot-aligned pre- and post-stow bin states from ARMBench snapshots via instance masks extraction and item matching. Stage 2 encodes bin states and stow context into latent vectors and then noisy tokens and condition tokens. Stage 3 uses a transformer-based latent diffusion model with cross-attention and adaptive layer normalization to incorporate condition tokens.
• Figure 4: Examples of post-stow bin state prediction (1st row: direct insert, 2nd row: sweep insert).
• Figure 5: Linear regression between DLO predictions obtained with ground-truth (GT) post-stow masks and those obtained from copy-paste with gravity (CP+G) or from FOREST.