Autonomous Field-of-View Adjustment Using Adaptive Kinematic Constrained Control with Robot-Held Microscopic Camera Feedback

TL;DR

The paper addresses FoV limitations in high-magnification microscopic robotics by introducing an autonomous camera automation framework that constrains a robot-held camera within the FoV while adapting the robot model, including camera extrinsics, using image-derived tooltip measurements. It combines a FoV-aware centralized kinematic controller with an adaptive loop that updates measurement-space parameters via a Jacobian-based formulation, leveraging a U-Net based tool tracker to feed the adaptation. The approach yields a substantial improvement in FoV maintenance, achieving 94.1% real FoV coverage in a bi-manual proof-of-concept compared to 54.4% with a non-adaptive baseline, and demonstrates how online extrinsics adaptation reduces model mismatch effects. The work offers a practical impact for autonomous microscopic manipulation, enabling more robust, self-calibrating visual servoing under tight FoV constraints and potentially enabling more complex autonomous tasks at mm-scale with camera feedback.

Abstract

Robotic systems for manipulation in millimeter scale often use a camera with high magnification for visual feedback of the target region. However, the limited field-of-view (FoV) of the microscopic camera necessitates camera motion to capture a broader workspace environment. In this work, we propose an autonomous robotic control method to constrain a robot-held camera within a designated FoV. Furthermore, we model the camera extrinsics as part of the kinematic model and use camera measurements coupled with a U-Net based tool tracking to adapt the complete robotic model during task execution. As a proof-of-concept demonstration, the proposed framework was evaluated in a bi-manual setup, where the microscopic camera was controlled to view a tool moving in a pre-defined trajectory. The proposed method allowed the camera to stay 94.1% of the time within the real FoV, compared to 54.4% without the proposed adaptive control.

Figures (5)

• Figure 1: The system configuration presented in this paper comprises two of the four arms from our AI-Robot Platform for Scientific Exploration marinho_design_2022. On the left is an arm holding a drill. On the right is an arm holding a camera. Above the workspace is a overview camera that provides stationary view of the workspace.
• Figure 2: Block diagram of the proposed system. The task-space controller receivesthe desired pose control signal to move R1 in a given trajectory, and both $R1$ and $R2$ to maintain the task constraints, including the FoV constraint. Meanwhile, a U-Net based tracking algorithm using images of the robot-mounted camera in $R2$ outputs the tooltip of the tool held by $R1$. Using these measurements, the adaptive controller updates the estimated parameters of the kinematic model of both robots, which include the camera extrinsics. That updated model is used by the task-space controller, and the task--adaptive loop is closed.
• Figure 3: Relevant elements for the camera extrinsics modeling of Section \ref{['sec:Proposed-camera-adaptive-methodo']}. Notice that $\boldsymbol{l}^{\text{oc}}\in\mathbb{H}_{p}\cap\mathbb{S}^{3}$ is the task-space measurement, the direction of the line connecting the pixel $\boldsymbol{\rho}^{\text{oc}}\left(u,v\right)\in\mathbb{I}^{2}$ representing the tooltip of $R1$ and the tooltip itself, passing through the optical center of the camera. Using this information, we can adapt the estimated robot model and extrinsics to compensate for initial modeling inaccuracies.
• Figure 4: Snapshots of the robot-mounted camera while $R1$ traverses a predefined trajectory. The drilltip frequently leaves the desired FoV when only kinematic constrained control is used, whereas it effectively stays within the desired FoV when using the proposed adaptive strategy.
• Figure 5: The plot trace the trajectories of the drill tip under the robot-mounted camera for two rotational period.