TriPilot-FF: Coordinated Whole-Body Teleoperation with Force Feedback

TL;DR

TriPilot-FF addresses the complexity of coordinating base mobility and bimanual manipulation by introducing a foot-operated pedal with lidar-driven resistive feedback, coupled with arm-side force reflection and real-time manipulability guidance to shape operator commands toward collision-averse behavior. The system integrates these interfaces with a bimanual leader-follower teleoperation stack and augments the ACT policy with joint-torque information, enabling torque-informed learning for whole-body tasks. Across real-world and simulated experiments, TriPilot-FF improves collision avoidance, reachability, and contact regulation, while torque-augmented ACT policies yield superior imitation-learning performance and better generalization to long-horizon and out-of-distribution scenarios. The open-source pedal hardware and software stack facilitate replication and future work on learning-enabled, multimodal teleoperation for mobile manipulators.

Abstract

Mobile manipulators broaden the operational envelope for robot manipulation. However, the whole-body teleoperation of such robots remains a problem: operators must coordinate a wheeled base and two arms while reasoning about obstacles and contact. Existing interfaces are predominantly hand-centric (e.g., VR controllers and joysticks), leaving foot-operated channels underexplored for continuous base control. We present TriPilot-FF, an open-source whole-body teleoperation system for a custom bimanual mobile manipulator that introduces a foot-operated pedal with lidar-driven pedal haptics, coupled with upper-body bimanual leader-follower teleoperation. Using only a low-cost base-mounted lidar, TriPilot-FF renders a resistive pedal cue from proximity-to-obstacle signals in the commanded direction, shaping operator commands toward collision-averse behaviour without an explicit collision-avoidance controller. The system also supports arm-side force reflection for contact awareness and provides real-time force and visual guidance of bimanual manipulability to prompt mobile base repositioning, thereby improving reach. We demonstrate the capability of TriPilot-FF to effectively ``co-pilot'' the human operator over long time-horizons and tasks requiring precise mobile base movement and coordination. Finally, we incorporate teleoperation feedback signals into an Action Chunking with Transformers (ACT) policy and demonstrate improved performance when the additional information is available. We release the pedal device design, full software stack, and conduct extensive real-world evaluations on a bimanual wheeled platform. The project page of TriPilot-FF is http://bit.ly/46H3ZJT.

Figures (15)

• Figure 1: TriPilot-FF is a whole-body teleoperation system that guides human operators via force feedback on both upper and lower limbs. We demonstrate the ability to operate over long-time horizons and in tasks requiring precise collision avoidance and bimanual coordination. Data from TriPilot-FF is used to train a policy that can robustly complete tasks which demand careful whole-body coordination.
• Figure 2: TriPilot-FF combines bimanual leader-follower arms with a pedal that commands the mobile base. The low-cost pedal has 3 DoF, and the design is shown on the left. An overview of the system is shown on the right. TriPilot-FF provides force feedback on both the operator-side leader arms and pedal based on contact, nearby obstacles, and to adjust the base for greater reachability.
• Figure 3: The potential $\phi$ produces an increasingly larger repulsive force as obstacles approach the robot.
• Figure 4: The manipulable regions highlighted over two wrist camera inputs. This informs the user whether an object is reachable without moving the base.
• Figure 5: Left: The bimanual manipulator repositions the bar vertically with both arms, before slipping through the narrow door, and then places the bar back horizontally. Right: The robot carries a box, which obstructs the view, and navigates through avoiding crate obstacles.