Robot Agnostic Visual Servoing considering kinematic constraints enabled by a decoupled network trajectory planner structure

TL;DR

The paper tackles visuospatial control for robotic manipulation by enforcing kinematic constraints within a robot-agnostic framework. It introduces a decoupled architecture: a task-specific Detection Network (DN) estimates target direction and orientation, while a Velocity Trajectory Planner (TP) generates a kinematically compliant velocity profile in Cartesian space. The key contributions are a robust, transfer-friendly VS method that respects robot limits, enables cross-robot deployment without retraining the detector, and demonstrates high precision (sub-0.5 mm position, sub-1° orientation) across two different robotic systems. The work shows practical impact by enabling reliable visual servoing in cluttered scenes and paves the way for 6-DoF extensions and task-aware trajectory optimization.

Abstract

We propose a visual servoing method consisting of a detection network and a velocity trajectory planner. First, the detection network estimates the objects position and orientation in the image space. Furthermore, these are normalized and filtered. The direction and orientation is then the input to the trajectory planner, which considers the kinematic constrains of the used robotic system. This allows safe and stable control, since the kinematic boundary values are taken into account in planning. Also, by having direction estimation and velocity planner separated, the learning part of the method does not directly influence the control value. This also enables the transfer of the method to different robotic systems without retraining, therefore being robot agnostic. We evaluate our method on different visual servoing tasks with and without clutter on two different robotic systems. Our method achieved mean absolute position errors of <0.5 mm and orientation errors of <1°. Additionally, we transferred the method to a new system which differs in robot and camera, emphasizing robot agnostic capability of our method.

Figures (7)

• Figure 1: Overview of the method. The task specific objects define the detection network used. The decoupled trajectory planner is parameterized by the kinematic constrains of the robot system.
• Figure 2: Results of the visual servoing task. First row shows the task solved with the UR5e robot, second row with the LR Mate 200iD/7L. Each column depicts a different initial alignment setting in the normal and clutter scene. The red path describes the trajectory of the detected OBB.
• Figure 3: Experimental setup of the visual servoing task, consisting of (1) UR5e robot, (2) D435 camera, (3) LR Mate 200iD/7L, (4) acA2040-35gc and (5) task setting, here clutter.
• Figure 4: Initial errors used for the experiments. Black indicates small initial error, gray large initial error and blue the target position. Next to the positions, the initial rotation error is written.
• Figure 5: Progression of the dsub connector visual servoing task using the UR5e with large initial error. (a) detected normalized $x$-direction, (b) detected normalized $y$-direction, (c) detected normalized orientation, (d) pixel error between OBB center and target $\xi$, (e) position error of the robot end effector and (f) the orientation error of the robot end effector.