Context-Aware Adaptive Shared Control for Magnetically-Driven Bimanual Dexterous Micromanipulation

Abstract

Magnetically actuated robots provide a promising untethered platform for navigation in confined environments, enabling biological studies and targeted micro-delivery. However, dexterous manipulation in complex structures remains challenging. While single-arm magnetic actuation suffices for simple transport, steering through tortuous or bifurcating channels demands coordinated control of multiple magnetic sources to generate the torques required for precise rotation and directional guidance. Bimanual teleoperation enables such dexterous steering but imposes high cognitive demands, as operators must handle the nonlinear dynamics of magnetic actuation while coordinating two robotic manipulators. To address these limitations, we propose Bi-CAST, a context-aware adaptive shared control framework for bimanual magnetic micromanipulation. A multimodal network fuses spatio-temporal visual features, spatial risk metrics, and historical states to continuously adjust the control authority of each manipulator in real time. In parallel, a bidirectional haptic interface integrates force-based intent recognition with risk-aware guidance, enabling force feedback to provide a continuous channel for dynamic human-machine authority negotiation. We validate the framework through user studies with eight participants performing three navigation tasks of increasing complexity in a vascular phantom. Compared with fixed authority and discrete switching baselines, Bi-CAST achieves up to 76.6% reduction in collisions, 25.9% improvement in trajectory smoothness, and 44.4% lower NASA-TLX workload, while delivering the fastest task completion times.

Figures (7)

• Figure 1: Overview of the bimanual shared micromanipulation framework, integrating human intent signals and autonomous path planning via a dynamic blending factor $\alpha$ predicted by a spatio-temporal network based on sequential video frames, enabling continuous and independent authority allocation.
• Figure 2: Overview of the bimanual micromanipulation platform. Two micromanipulators are arranged in an opposing configuration, each tipped with a magnet for millirobot actuation. The operator controls the system through two haptic devices equipped with FSR sensors for user intent capture. A digital microscope provides real-time imaging.
• Figure 3: Architecture of the multimodal authority prediction network. Three input streams (RGB frames, bilateral FSR force features, and safety metrics) are encoded and fused through a Transformer encoder with eight self-attention blocks. The network outputs bilateral authority chunks of five future steps, which are causally aggregated via exponential temporal weighting: $\alpha_t^i = \sum_{k=0}^{C-1} \omega_k \hat{\alpha}_{t-k,i,k}$.
• Figure 4: Dual-arm path planning across three tasks of increasing complexity: Target A (one bifurcation), Target B (additional bifurcations), and Target C (multiple bifurcations with narrowing channels). The heatmap indicates distance-transform-based travel cost, and overlaid poses show the millirobot's orientation changes along each path.
• Figure 5: Millirobot localization under partial occlusion. (a) Raw microscope image with the manipulator tip occluding the millirobot. (b) YOLO segmentation mask overlaid with tracking result, visualizing IoU. (c) Robust localization maintained via frame-to-frame registration-based tracking.