Camera Frame Misalignment in a Teleoperated Eye-in-Hand Robot: Effects and a Simple Correction Method

TL;DR

This work investigates camera frame misalignment in teleoperated eye-in-hand robots, where camera orientation dynamically shifts with end-effector motion. It introduces a simple view-display correction that rotates the camera image about its $Z$ axis by a computed angle $ heta$, derived via $ heta = ext{atan2}(\mathbf{X}_T^T\mathbf{Y}_A-\mathbf{Y}_T^T\mathbf{X}_A, \mathbf{X}_T^T\mathbf{X}_A+\mathbf{Y}_T^T\mathbf{Y}_A)$, to align the operator's view with the intended motion. Through a simulated three-way mixed design and a validation on a real tubular eye-in-hand robot, the correction consistently reduces completion time, trajectory length, and workload metrics, with substantial percentage improvements (e.g., mean completion time reductions around 40–50% and subjective workload reductions above 50%). These results suggest that a simple, display-based correction can significantly enhance performance in compact, endoscopic-scale teleoperation, with practical implications for surgical robotics and other confined-environment applications. Future work will extend the correction to additional degrees of freedom and explore coupling effects with more complex pitch-yaw configurations.

Abstract

Misalignment between the camera frame and the operator frame is commonly seen in a teleoperated system and usually degrades the operation performance. The effects of such misalignment have not been fully investigated for eye-in-hand systems - systems that have the camera (eye) mounted to the end-effector (hand) to gain compactness in confined spaces such as in endoscopic surgery. This paper provides a systematic study on the effects of the camera frame misalignment in a teleoperated eye-in-hand robot and proposes a simple correction method in the view display. A simulation is designed to compare the effects of the misalignment under different conditions. Users are asked to move a rigid body from its initial position to the specified target position via teleoperation, with different levels of misalignment simulated. It is found that misalignment between the input motion and the output view is much more difficult to compensate by the operators when it is in the orthogonal direction (~40s) compared with the opposite direction (~20s). An experiment on a real concentric tube robot with an eye-in-hand configuration is also conducted. Users are asked to telemanipulate the robot to complete a pick-and-place task. Results show that with the correction enabled, there is a significant improvement in the operation performance in terms of completion time (mean 40.6%, median 38.6%), trajectory length (mean 34.3%, median 28.1%), difficulty (50.5%), unsteadiness (49.4%), and mental stress (60.9%).

Figures (15)

• Figure 1: An example of camera frame misalignment in a teleoperated eye-in-hand robot. a) A tubular robot with a camera mounted to the end-effector. The image returned by the camera is not consistent with the operator's view as the camera is rotated about its axis. b) When the operator applies a motion in the left direction (as indicated by the pink arrow on the operand), the camera also moves left (as indicated by the pink arrow on the camera), and the objects in the image should be viewed as if they move right (as indicated by the pink arrow in the corrected camera view) if no misalignment exists. However, the actual image moves downwards (as indicated by the pink arrow in the actual cameral view) due to misalignment of the actual camera frame {$X_A$, $Y_A$, $Z_A$} with respect to the teleoperator frame {$X_T$, $Y_T$, $Z_T$}.
• Figure 2: A simple correction method for the camera frame misalignment. Frames {T}, {A}, and {C} represent the teleoperator's, the actual, and the corrected frame, respectively. The corrected frame is obtained by rotating the actual frame about its $\mathbf{Z}$ axis for an angle of $\theta$.
• Figure 3: Teleoperation experiments in a simulated environment. a) Experimental setup. The operator uses a 3D joystick to manipulate a quadruplet constituted by a sphere and three cubes in a MATLAB simulation. The task is to move the camera from the initial position to a position where the 2D projection of the sphere lies between the two concentric circles at the center of the camera view. b) - i) Example images presented to the operator under different conditions. Note that the states between WOC and WC when $ROLL=0^\circ,PITCH=0^\circ,YAW=0^\circ$ are the same as there is no misalignment under this condition.
• Figure 4: Block diagram of the simulation algorithm. The orientation of the camera is determined by ROLL, PITCH, and YAW angles. Two conditions, WOC and WC, can be selected through a switch by the experimenter. The simulation runs in Simulink (9.0) 3D Animation on a Windows 10 platform with Intel Core i5-7300U CPU @ 2.60GHz 2.71GHz and 8.00GB RAM.
• Figure 5: The RPY representation used in this paper. The three rotations transform the original frame $X_0Y_0Z_0$ to frame $X_3Y_3Z_3$ following the shown order. As the $\mathbf{Z}$ axis is located along the camera, it can be seen that the YAW and PITCH angles determine the direction of the axis of the camera, while the ROLL angle describes the rotation of the camera about its own axis.

Theorems & Definitions (3)

• Remark 1
• Remark 2
• Remark 3