Shape-Interpretable Visual Self-Modeling Enables Geometry-Aware Continuum Robot Control

TL;DR

A shape-interpretable visual self-modeling framework for continuum robots that enables geometry-aware control and advances autonomous, geometry-aware manipulation for continuum robots is introduced.

Abstract

Continuum robots possess high flexibility and redundancy, making them well suited for safe interaction in complex environments, yet their continuous deformation and nonlinear dynamics pose fundamental challenges to perception, modeling, and control. Existing vision-based control approaches often rely on end-to-end learning, achieving shape regulation without explicit awareness of robot geometry or its interaction with the environment. Here, we introduce a shape-interpretable visual self-modeling framework for continuum robots that enables geometry-aware control. Robot shapes are encoded from multi-view planar images using a Bezier-curve representation, transforming visual observations into a compact and physically meaningful shape space that uniquely characterizes the robot's three-dimensional configuration. Based on this representation, neural ordinary differential equations are employed to self-model both shape and end-effector dynamics directly from data, enabling hybrid shape-position control without analytical models or dense body markers. The explicit geometric structure of the learned shape space allows the robot to reason about its body and surroundings, supporting environment-aware behaviors such as obstacle avoidance and self-motion while maintaining end-effector objectives. Experiments on a cable-driven continuum robot demonstrate accurate shape-position regulation and tracking, with shape errors within 1.56% of image resolution and end-effector errors within 2% of robot length, as well as robust performance in constrained environments. By elevating visual shape representations from two-dimensional observations to an interpretable three-dimensional self-model, this work establishes a principled alternative to vision-based end-to-end control and advances autonomous, geometry-aware manipulation for continuum robots.

Figures (6)

• Figure 1: Conceptual illustration of the bio-inspired self-modeling mechanism. (A) Self-learning mechanism in biological systems. An elephant controls the motion of its trunk through muscular actuation and acquires corresponding trunk shape feedback via binocular vision, thereby learning the relationship between muscle activation and trunk morphology. Based on this learned internal model, the elephant can anticipate the shape changes induced by its own motor commands and accordingly control the trunk to perform complex manipulation tasks. (B) The continuum robot is actuated using arbitrary temporal input signals, while image data of the robot are recorded at each time instant from two different views. The image data are parameterized and encoded together with the actuation signals to form a dataset for neural network training. After training, the continuum robot can exploit the learned network to autonomously predict the visual shape dynamics from different views under given input temporal signals.
• Figure 2: Shape-interpretable visual self-modeling and geometry-aware control framework for continuum robots. (A) The robot shape is represented using a piecewise Bézier curve, which captures global shape characteristics with a small number of control points. (B) The robot region is first extracted using image processing techniques, and the skeleton curve is obtained via morphological operations. The skeleton is then fitted by Bézier curves, and the resulting control points are used as shape parameters, enabling an interpretable shape encoding. (C) Position NODE and shape NODE models are trained using the collected shape, position, and actuation data to achieve shape-position self-modeling. (D) When an environmental obstacle enters a predefined warning distance, the obstacle avoidance mechanism is activated. A repulsive escape velocity is applied at the point on the robot body closest to the obstacle, and this local escape velocity is mapped to a global shape variation, thereby realizing obstacle avoidance through shape control. (E) The trained position NODE and shape NODE are used to estimate the position Jacobian and shape Jacobian, respectively. A hybrid controller is then constructed by integrating the position controller, shape controller, and obstacle avoidance controller. (F) The proposed control framework is validated on a three-segment continuum robot and is capable of performing shape-position regulation, shape-position tracking, obstacle-aware regulation, and self-motion tasks.
• Figure 3: Experimental results of continuum robot shape-position regulation using the proposed method. (A) Experimental snapshots from two different views during the process in which the proposed method regulates the robot to the reference shape and end-effector position. (B) The robot end-effector moves from the initial position to the reference position. (C) Convergence of the shape-state regulation error to zero. (D) Convergence of the end-effector position regulation error to zero. (E) Four different shape-position regulation tasks are all successfully accomplished.
• Figure 4: Experimental results of simultaneous shape-position tracking control of a continuum robot using the proposed method. (A) Experimental snapshots from two different views during tracking of the $\infty$-shaped end-effector trajectory. (B) Profile of the robot end-effector trajectory during tracking of the $\infty$-shaped trajectory. (C) Experimental snapshots from two different views during tracking of the 8-shaped end-effector trajectory. (D) Profile of the robot end-effector trajectory during tracking of the 8-shaped trajectory. (E) Distribution of shape tracking errors from different views. (F) Distribution of tracking errors along different coordinate axes.
• Figure 5: Experimental results of shape-position control of a continuum robot in an environment with obstacle using the proposed method. (A) Experimental snapshots from two different views during shape-position regulation in the obstacle environment. (B) Evolution of the robot end-effector position during the shape-position regulation task in the presence of obstacle. (C) Minimum visual distance between the robot body and the obstacle. (D) Shape-state regulation error during the shape-position regulation task in the obstacle environment. (E) End-effector position regulation error during the shape-position regulation task in the obstacle environment. (F) The robot avoids a dynamic obstacle by shape control while keeping the end-effector position unchanged.