Going with the Flow: Koopman Behavioral Models as Implicit Planners for Visuo-Motor Dexterity

TL;DR

The paper introduces Unified Behavioral Models (UBMs) to represent dexterous visuo-motor skills as coupled robot–environment dynamics, addressing the temporal incoherence of reactive policies. Koopman-UBM leverages a state-inclusive lifting and a linear latent evolution $z_{t+1}=K z_t$ to act as an implicit visuo-motor planner, trained with a multi-step coherence objective and stabilized by identity initialization and separate learning rates. Visual features ground the latent state via two pipelines: Object Flow points and DynaMo, enabling reliable long-horizon predictions and real-time replanning through event-triggered triggers when predicted visual flow diverges from observations. Across seven simulated and two real-world tasks, K-UBM achieves state-of-the-art or near-state-of-the-art performance with faster inference, robustness to occlusion, and flexible replanning, while offering insights into robust Koopman learning and open-loop planning in dexterous manipulation.

Abstract

There has been rapid and dramatic progress in learning complex visuo-motor manipulation skills from demonstrations, thanks in part to expressive policy classes that employ diffusion- and transformer-based backbones. However, these design choices require significant data and computational resources and remain far from reliable, particularly within the context of multi-fingered dexterous manipulation. Fundamentally, they model skills as reactive mappings and rely on fixed-horizon action chunking to mitigate jitter, creating a rigid trade-off between temporal coherence and reactivity. In this work, we introduce Unified Behavioral Models (UBMs), a framework that learns to represent dexterous skills as coupled dynamical systems that capture how visual features of the environment (visual flow) and proprioceptive states of the robot (action flow) co-evolve. By capturing such behavioral dynamics, UBMs can ensure temporal coherence by construction rather than by heuristic averaging. To operationalize these models, we propose Koopman-UBM, a first instantiation of UBMs that leverages Koopman Operator theory to effectively learn a unified representation in which the joint flow of latent visual and proprioceptive features is governed by a structured linear system. We demonstrate that Koopman-UBM can be viewed as an implicit planner: given an initial condition, it computes the desired robot behavior with the resulting flow of visual features over the entire skill horizon. To enable reactivity, we introduce an online replanning strategy in which the model acts as its own runtime monitor that automatically triggers replanning when predicted and observed visual flow diverge. Across seven simulated and two real-world tasks, we demonstrate that K-UBM matches or exceeds the performance of SOTA baselines, while offering faster inference, smooth execution, robustness to occlusions, and flexible replanning.

Figures (11)

• Figure 1: (A) Standard reactive policies (e.g., Diffusion, ACT) map observations to short-horizon action chunks, lacking a consistent internal model of the future or memory beyond the observation window, leading to temporal incoherence and hand-coded chunk lengths. (B) In contrast, Unified Behavioral Models (UBM) model skills as joint behavioral dynamics of the robot and environment governing a continuous flow in a latent space, ensuring coherence and enabling full-horizon "planning" from initial conditions. (C) We propose Koopman-UBM as the first instantiation of UBMs, which lifts visual and proprioceptive observations into a latent space governed by a learned Koopman Operator. By enforcing linear spectral dynamics ($z_{t+1} = Kz_t$) over a unified "state-inclusive" latent space, K-UBM ensures enables fast inference and predictive monitoring. (D) Our approach enables temporally-coherent predictions (dashed-purple) that are robust to visual occlusion (dashed green and purple trajectories inside gray boxes) and reactivity via an event-triggered replanning strategy that reinitializes the UBM only when the predicted visual features diverge from reality (top orange box).
• Figure 2: We evaluate K-UBM on seven simulations tasks and two real-world tasks.
• Figure 3: We report the task success rates for each method on the test sets using both visual features (Flow on the top row and DynaMo on the bottom row). For each method, error bars indicate the standard deviation over five random seeds, and average results (last column) are computed over all seeds and all tasks. ACT (10, 20, 40) denotes three ACT variants with different prediction horizons, while Diffusion (2, 4, 8) and (2, 8, 16) denote different execution and prediction horizons.
• Figure 4: We visualize the predicted flow points by decoding the flow features and overlaying them on the corresponding images.
• Figure 5: We visualize the predicted DynaMo features by projecting them together with the ground-truth features using t-SNE.