Inference-Time Policy Steering through Human Interactions

TL;DR

This work addresses the challenge of steering pre-trained, multimodal generative policies at inference time without fine-tuning. It introduces Inference-Time Policy Steering (ITPS), a framework that biases diffusion-based sampling using human interactions (point, sketch, physical correction) to align outputs with user intent while preserving in-distribution validity. By formalizing alignment metrics (Task Alignment $TA$, Motion Alignment $MA$) and constraint satisfaction $CS$, and evaluating six steering methods, the study finds stochastic sampling with diffusion policies offers the best balance between alignment and distribution shift across Maze2D, Block Stacking, and Real World Kitchen tasks. The findings demonstrate the potential to adapt generalist policies to downstream human goals in real time, with implications for safer, more controllable robot behavior without retraining, albeit at a computational cost. $TA$, $MA$, and $CS$ provide a principled lens to trade off user intent satisfaction against constraint adherence in inference-time steering.$

Abstract

Generative policies trained with human demonstrations can autonomously accomplish multimodal, long-horizon tasks. However, during inference, humans are often removed from the policy execution loop, limiting the ability to guide a pre-trained policy towards a specific sub-goal or trajectory shape among multiple predictions. Naive human intervention may inadvertently exacerbate distribution shift, leading to constraint violations or execution failures. To better align policy output with human intent without inducing out-of-distribution errors, we propose an Inference-Time Policy Steering (ITPS) framework that leverages human interactions to bias the generative sampling process, rather than fine-tuning the policy on interaction data. We evaluate ITPS across three simulated and real-world benchmarks, testing three forms of human interaction and associated alignment distance metrics. Among six sampling strategies, our proposed stochastic sampling with diffusion policy achieves the best trade-off between alignment and distribution shift. Videos are available at https://yanweiw.github.io/itps/.

Figures (11)

• Figure 1: Inference-Time Policy Steering (ITPS). We present a novel framework to unify various forms of human interactions to steer a frozen generative policy. User interactions "prompt" pre-trained policies to synthesize aligned behaviors at inference time.
• Figure 2: Policy Steering Methods. Given user input, methods (a-c) incorporate the alignment objective either before or after inference via (a) perturbation, (b) ranking, or (c) initialization, whereas methods (d,e) integrate the objective directly during inference.
• Figure 3: Guided Diffusion vs. Stochastic Sampling. In a toy example aiming to sample likely data points from a pre-trained distribution while aligning with a target point, GD samples approximate the sum of two distributions, whereas SS samples approximate their product, as illustrated by contour lines from kernel density estimation Waskom2021. Consequently, when the point input does not align with any distribution mode, GD introduces distribution shift, while SS identifies the closest in-distribution mode.
• Figure 4: Alignment vs. Collision in Maze2D. We compare various sampling methods with ACT and DP steered using sketch input. (1) Steering frozen policies improves alignment at the cost of constraint satisfaction and increased collisions. Moreover, (2) Multimodal policies (DP) steered with PR enhance alignment without significant distribution shift, while (3) unimodal policies (ACT) are harder to steer effectively, particularly if they lack robustness (see Figure \ref{['fig:maze_csr']}). (4) Finally, DP steered with SS achieves the best alignment-constraint satisfaction trade-off.
• Figure 5: Maze2D Qualitative Comparisons. We visualize trajectories (color-coded from blue to red over time) sampled with various steering methods from two policy classes (ACT and DP) given a sketch in gray. Trajectory thickness reflects similarity to the sketch after ranking, and samples in collision are tinted white. SS preserves collision-free constraints while aligning with user intent.