Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Compositional Visual Planning via Inference-Time Diffusion Scaling

Yixin Zhang, Yunhao Luo, Utkarsh Aashu Mishra, Woo Chul Shin, Yongxin Chen, Danfei Xu

TL;DR

This work formulates long-horizon planning as inference over a chain-structured factor graph of overlapping video chunks, where pretrained short-horizon video diffusion models provide local priors, and enforce boundary agreement through a novel combination of synchronous and asynchronous message passing that operates on Tweedie estimates.

Abstract

Diffusion models excel at short-horizon robot planning, yet scaling them to long-horizon tasks remains challenging due to computational constraints and limited training data. Existing compositional approaches stitch together short segments by separately denoising each component and averaging overlapping regions. However, this suffers from instability as the factorization assumption breaks down in noisy data space, leading to inconsistent global plans. We propose that the key to stable compositional generation lies in enforcing boundary agreement on the estimated clean data (Tweedie estimates) rather than on noisy intermediate states. Our method formulates long-horizon planning as inference over a chain-structured factor graph of overlapping video chunks, where pretrained short-horizon video diffusion models provide local priors. At inference time, we enforce boundary agreement through a novel combination of synchronous and asynchronous message passing that operates on Tweedie estimates, producing globally consistent guidance without requiring additional training. Our training-free framework demonstrates significant improvements over existing baselines, effectively generalizing to unseen start-goal combinations that were not present in the original training data. Project website: https://comp-visual-planning.github.io/

Compositional Visual Planning via Inference-Time Diffusion Scaling

TL;DR

This work formulates long-horizon planning as inference over a chain-structured factor graph of overlapping video chunks, where pretrained short-horizon video diffusion models provide local priors, and enforce boundary agreement through a novel combination of synchronous and asynchronous message passing that operates on Tweedie estimates.

Abstract

Diffusion models excel at short-horizon robot planning, yet scaling them to long-horizon tasks remains challenging due to computational constraints and limited training data. Existing compositional approaches stitch together short segments by separately denoising each component and averaging overlapping regions. However, this suffers from instability as the factorization assumption breaks down in noisy data space, leading to inconsistent global plans. We propose that the key to stable compositional generation lies in enforcing boundary agreement on the estimated clean data (Tweedie estimates) rather than on noisy intermediate states. Our method formulates long-horizon planning as inference over a chain-structured factor graph of overlapping video chunks, where pretrained short-horizon video diffusion models provide local priors. At inference time, we enforce boundary agreement through a novel combination of synchronous and asynchronous message passing that operates on Tweedie estimates, producing globally consistent guidance without requiring additional training. Our training-free framework demonstrates significant improvements over existing baselines, effectively generalizing to unseen start-goal combinations that were not present in the original training data. Project website: https://comp-visual-planning.github.io/
Paper Structure (57 sections, 4 theorems, 23 equations, 17 figures, 5 tables, 1 algorithm)

This paper contains 57 sections, 4 theorems, 23 equations, 17 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Diffusion Models For Planning.
  4. Compositional Diffusion Generation.
  5. Inference-Time Guidance for Diffusion Model.
  6. Preliminaries
  7. Factor Graph Formulation for Compositional Distributions
  8. Diffusion model and Training-Free Guided Diffusion
  9. Method
  10. Problem formulation
  11. Distribution of Factor Graph
  12. Interpretation.
  13. Jointly Synchronous and Asynchronous Message Passing
  14. Synchronous Message Passing
  15. Asynchronous Message Passing
  16. ...and 42 more sections

Key Result

Theorem 1

Consider a linear chain $z=[u^1,u^2,u^3]$ with pairwise factors $[u^1,u^2]$ and $[u^2,u^3]$, where $u^2$ is the transition boundary variable. Assume the forward noising processes are $p(u_t^1,u_t^2\,|\,u^1,u^2)$, $p(u_t^2,u_t^3\,|\,u^2,u^3)$, and $p(u_t^2\,|\,u^2)$. Let $a(u^2)= \int p(u^1, u^2) p(u

Figures (17)

  • Figure 1: Compositional Visual Planning via Inference Time Diffuser Scaling. We train a short-horizon visual diffusion model on clips treated as a single factor. At inference, we scale visual planning horizon without retraining by chaining overlapping factors into a linear factor graph: the start and goal boundary variables are anchored at the ends, while neighboring factors exchange information through shared transition boundary variables.
  • Figure 2: Motivating toy example. We train a short-horizon diffusion model on circular arc clips (left). At test time, three $120^\circ$ arc generators are composed to form a three-petal “flower”.
  • Figure 3: Real World Task Layout.This task involves 2 start and 2 goal configurations. We also evaluate on more challenging tasks. A complete list of task settings is provided in Appendix \ref{['appendix:benchmark_tasks']}.
  • Figure 4: Effect of synchronous and asynchronous message passing.
  • Figure 5: Effect of sampling steps on planning performance.
  • ...and 12 more figures

Theorems & Definitions (5)

  • Theorem 1: Noisy-Bethe Gap Theorem
  • Proposition 1: Synchronous Message Passing Constraint
  • Theorem 1: Noisy-Bethe Gap Theorem
  • proof
  • Proposition 1: Synchronous Message Passing Constraint