Learning From a Steady Hand: A Weakly Supervised Agent for Robot Assistance under Microscopy

TL;DR

This work tackles microscope-guided robotic micromanipulation by reframing steady-hand co-manipulation as a weakly supervised learning problem. It introduces a two-stage 3D perception pipeline (lateral tip detection plus depth estimation) trained from warm-up demonstrations, paired with an uncertainty-aware, markerless hand-eye calibration and a macro-micro fusion controller. The approach yields calibrated, depth-resolved guidance without external fiducials, and demonstrates significant workload reductions ($ ext{NASA-TLX}$) in a within-subject user study while maintaining sub-millimeter lateral and sub-millimeter-to-sub-millimeter axial accuracy. The framework provides a practical, safety-aware pathway to deploy microscope-guided interventions by explicitly budgeting uncertainty and enabling robust, real-time control under monocular vision.

Abstract

This paper rethinks steady-hand robotic manipulation by using a weakly supervised framework that fuses calibration-aware perception with admittance control. Unlike conventional automation that relies on labor-intensive 2D labeling, our framework leverages reusable warm-up trajectories to extract implicit spatial information, thereby achieving calibration-aware, depth-resolved perception without the need for external fiducials or manual depth annotation. By explicitly characterizing residuals from observation and calibration models, the system establishes a task-space error budget from recorded warm-ups. The uncertainty budget yields a lateral closed-loop accuracy of approx. 49 micrometers at 95% confidence (worst-case testing subset) and a depth accuracy of <= 291 micrometers at 95% confidence bound during large in-plane moves. In a within-subject user study (N=8), the learned agent reduces overall NASA-TLX workload by 77.1% relative to the simple steady-hand assistance baseline. These results demonstrate that the weakly supervised agent improves the reliability of microscope-guided biomedical micromanipulation without introducing complex setup requirements, offering a practical framework for microscope-guided intervention.

Figures (11)

• Figure 1: The data collection relies on "warm-up" episodes: short trajectories generated by the operator dragging the tool under steady-hand guidance. These maneuvers are standardized into two tasks: lateral sweeps for FOV coverage (the red line), and axial scans for controlled depth-axis motion (the orange line). The agent leverages warm-up trajectories to train the 3D tip observer and estimate hand-eye fusion parameters. Leveraging the dataset-learned calibration and tip recognition, the agent servos laterally to the operator-designated target (e.g., for aspiration) and regulates axial depth for station-keeping.
• Figure 2: To circumvent of dense manual labeling, we utilize "steady-hand" warm-up clips. By manually prompting SAM 2 on only a few anchor frames (sparse annotation), the system propagates tip masks bidirectionally to generate dense, real-time 3D tip estimation targets.
• Figure 3: We deploy a two-stage detector for real-time 3D tip estimation. The tip estimation block (TEB) returns the keypoint observations and the depth estimation block (DEB) returns the normalized depth from tip to the focal plane and an auxiliary classification label.
• Figure 4: Dataset acquisition setup. (a) Upright microscope. (b-c) Inverted microscope with illumination control and a linear module for camera motion. (d-e) Steady-hand data collection under both setups.
• Figure 5: Robotic micromanipulation setup. A Meca500 arm with a loadcell operates under an inverted microscope. The system integrates manual macro-guidance and automated visual servoing.