Whole-Body Safe Control of Robotic Systems with Koopman Neural Dynamics

TL;DR

A data-driven framework is proposed that learns a Koopman embedding and operator from data, and integrates the resulting linear model with the Safe Set Algorithm, which allows the tracking and safety constraints to be solved in a single quadratic program (QP), ensuring feasibility and optimality without a separate safety filter.

Abstract

Controlling robots with strongly nonlinear, high-dimensional dynamics remains challenging, as direct nonlinear optimization with safety constraints is often intractable in real time. The Koopman operator offers a way to represent nonlinear systems linearly in a lifted space, enabling the use of efficient linear control. We propose a data-driven framework that learns a Koopman embedding and operator from data, and integrates the resulting linear model with the Safe Set Algorithm (SSA). This allows the tracking and safety constraints to be solved in a single quadratic program (QP), ensuring feasibility and optimality without a separate safety filter. We validate the method on a Kinova Gen3 manipulator and a Go2 quadruped, showing accurate tracking and obstacle avoidance.

Figures (8)

• Figure 1: Left: Kinova Gen3 safely moving from START (red) to GOAL (green) while avoiding the collision volume (yellow); collision avoidance is enforced using the learned Koopman dynamics within our unified Koopman--SSA MPC. Middle: Floating-base safe control in Isaac simulation (Unitree Go2). The learned Koopman-MPC tracks the reference while actively avoiding the obstacle-induced unsafe region; collision avoidance is enforced using the learned Koopman dynamics within the MPC rollout. Right: We use the learned Koopman lifted space and its linear dynamics (blue) to predict motion in MPC and evaluate the safety constraints, which enables the collision avoidance shown in the left and middle panels.
• Figure 3: Overview of training framework for Koopman Safe Control. Neural embedding function and Koopman Operators are first trained, safety index is adapted to the learned dynamics via adversarial fine tuning, and the model is migrated to real environment.
• Figure 4: Prediction Error Comparison. The analytic baselines (LTI and LTV), KDM, and NNDM are compared against PyBullet ground truth. KDM maintains the lowest long-horizon error growth, while the analytic model is competitive in short horizons but diverges.
• Figure 5: Safe Tracking Comparison. NMPC-10 is nonlinear MPC with 10 times shooting, values in parentheses represent slack weight for safety filter relaxation. Linear counterparts, including KMPC did not require slack relaxation for this experiment. KMPC outperforms NMPC-10 both in tracking and safety constraint satisfaction.
• Figure 6: Histogram of single-step joint-angle prediction error norms (radians) over five hardware trajectories (rectangle, sin, star, triangle, spiral), comparing the original Koopman model (blue) against the fine-tuned Koopman model (orange). Fine-tuning shifts the error distributions toward smaller values across all trajectories, yielding a mean joint-angle error of 0.140 rad (maximum 0.546 rad) and a mean end-effector position error of 0.031 m (maximum 0.143 m) on the evaluated hardware dataset.