Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Inverse Kinematics with Vision-Based Constraints

Liangting Wu, Roberto Tron

TL;DR

This work defines Visual Inverse Kinematics (VIK), aiming to compute robot configurations that satisfy both kinematic constraints and visibility of a target from an on-board camera. It encodes the visibility constraint by introducing a virtual chain and solves the resulting rank-constrained problem through a semidefinite programming (SDP) relaxation followed by a rank-minimization step, ensuring tractable optimization with local convergence guarantees. The approach supports several vision-based objectives, including levelness, centering, and reprojection accuracy, which can be combined linearly within the optimization. Demonstrations on a 7-DOF Sawyer manipulator with a hand-mounted camera show the method can enforce FoV visibility while achieving different visual goals, highlighting practical potential for initializing or guiding visual servo tasks. Overall, the method provides a principled, convex-relaxation–plus–rank-minimization pipeline to integrate vision constraints into IK-like planning with robust convergence properties.

Abstract

This paper introduces the Visual Inverse Kinematics problem (VIK) to fill the gap between robot Inverse Kinematics (IK) and visual servo control. Different from the IK problem, the VIK problem seeks to find robot configurations subject to vision-based constraints, in addition to kinematic constraints. In this work, we develop a formulation of the VIK problem with a Field of View (FoV) constraint, enforcing the visibility of an object from a camera on the robot. Our proposed solution is based on the idea of adding a virtual kinematic chain connecting the physical robot and the object; the FoV constraint is then equivalent to a joint angle kinematic constraint. Along the way, we introduce multiple vision-based cost functions to fulfill different objectives. We solve this formulation of the VIK problem using a method that involves a semidefinite program (SDP) constraint followed by a rank minimization algorithm. The performance of this method for solving the VIK problem is validated through simulations.

Inverse Kinematics with Vision-Based Constraints

TL;DR

This work defines Visual Inverse Kinematics (VIK), aiming to compute robot configurations that satisfy both kinematic constraints and visibility of a target from an on-board camera. It encodes the visibility constraint by introducing a virtual chain and solves the resulting rank-constrained problem through a semidefinite programming (SDP) relaxation followed by a rank-minimization step, ensuring tractable optimization with local convergence guarantees. The approach supports several vision-based objectives, including levelness, centering, and reprojection accuracy, which can be combined linearly within the optimization. Demonstrations on a 7-DOF Sawyer manipulator with a hand-mounted camera show the method can enforce FoV visibility while achieving different visual goals, highlighting practical potential for initializing or guiding visual servo tasks. Overall, the method provides a principled, convex-relaxation–plus–rank-minimization pipeline to integrate vision constraints into IK-like planning with robust convergence properties.

Abstract

This paper introduces the Visual Inverse Kinematics problem (VIK) to fill the gap between robot Inverse Kinematics (IK) and visual servo control. Different from the IK problem, the VIK problem seeks to find robot configurations subject to vision-based constraints, in addition to kinematic constraints. In this work, we develop a formulation of the VIK problem with a Field of View (FoV) constraint, enforcing the visibility of an object from a camera on the robot. Our proposed solution is based on the idea of adding a virtual kinematic chain connecting the physical robot and the object; the FoV constraint is then equivalent to a joint angle kinematic constraint. Along the way, we introduce multiple vision-based cost functions to fulfill different objectives. We solve this formulation of the VIK problem using a method that involves a semidefinite program (SDP) constraint followed by a rank minimization algorithm. The performance of this method for solving the VIK problem is validated through simulations.
Paper Structure (17 sections, 2 theorems, 18 equations, 5 figures, 1 table, 1 algorithm)

This paper contains 17 sections, 2 theorems, 18 equations, 5 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Kinematic constraints Using Rank-1 Semidefinite Matrices
  3. Revolute joints
  4. Prismatic Joints
  5. Inverse kinematics and rank-constrained optimization
  6. Visual Inverse Kinematics
  7. Perspective projection and virtual prismatic joints
  8. Visibility constraint
  9. Vision-based costs
  10. Levelness
  11. Centering
  12. Minimizing the reprojection error
  13. Optimization
  14. Convex relaxation
  15. Rank minimization
  16. ...and 2 more sections

Key Result

Theorem 1

The set $\mathcal{U}$ is a subset of $\bar{\mathcal{U}}$, and is equal to the intersection of $\bar{\mathcal{U}}$ and $\mathcal{R}_1$, i.e., $\mathcal{U}=\bar{\mathcal{U}}\cap \mathcal{R}_1$, where $\mathcal{R}_1$ is the set of $\mathbf{u}=g(\mathbf{Y})\in\mathbb R^{9 n_r}$ such that each $\mathbf{Y

Figures (5)

  • Figure 1: The field of view is represented as a right circular cone whose axis is aligned with the camera Z-axis.
  • Figure 2: The object is connected with the camera center through a virtual chain consisted of prismatic joints and spherical joints. Each prismatic joint is parameterized with two reference frames sharing the same $z$-axis.
  • Figure 3: The visibility constraint formulated as a ball bound on the difference $\mathbf{R}_i^{(3)}-\mathbf{R}^{(3)}_{\textrm{c}}$.
  • Figure 4: Our solver finds a posture of Sawyer subject to the visibility constraint of capturing the quadrotors in the camera field of view while targeting different vision-based objectives.
  • Figure 5: The largest eigenvalues during the rank minimization process while solving the problem in Fig. \ref{['fig:quad1']} is increased to the fixed values of traces, i.e., $\mathop{\mathrm{tr}}\nolimits(\mathbf{Y}_i)=3$ and $\mathop{\mathrm{tr}}\nolimits(\mathbf{Y}_{\tau i})=2$, indicating rank-1 solutions.

Theorems & Definitions (6)

  • Remark 1
  • Definition 1
  • Theorem 1
  • Definition 2
  • Theorem 2
  • Remark 2