Beyond Visuals: Investigating Force Feedback in Extended Reality for Robot Data Collection

TL;DR

This work investigates how force feedback influences XR-based robot data collection, testing two control interfaces—Kinesthetic Teaching and VR Motion Controller—with and without force feedback across four manipulation tasks. Through a 31-participant within-subject study and 496 demonstrations, the authors show that force feedback generally enhances efficiency, effectiveness, and user experience, especially for complex, precision-demanding tasks like Assemble Peg, and that gains are more pronounced for the MC interface than for KT. A key contribution is the Kinesthetic Teaching with Force Feedback (KTF) interface, which transmits virtual joint forces to the physical robot to improve perceptual fidelity and performance. The findings suggest that force-feedback design should be tailored to task difficulty and interface modality to maximize data-collection quality and operator usability in real-world teleoperation scenarios, with implications for future XR-enabled robotic teleoperation systems.

Abstract

This work explores how force feedback affects various aspects of robot data collection within the Extended Reality (XR) setting. Force feedback has been proved to enhance the user experience in Extended Reality (XR) by providing contact-rich information. However, its impact on robot data collection has not received much attention in the robotics community. This paper addresses this shortcoming by conducting an extensive user study on the effects of force feedback during data collection in XR. We extended two XR-based robot control interfaces, Kinesthetic Teaching and Motion Controllers, with haptic feedback features. The user study is conducted using manipulation tasks ranging from simple pick-place to complex peg assemble, requiring precise operations. The evaluations show that force feedback enhances task performance and user experience, particularly in tasks requiring high-precision manipulation. These improvements vary depending on the robot control interface and task complexity. This paper provides new insights into how different factors influence the impact of force feedback.

Figures (5)

• Figure 1: Kinesthetic Teaching (KT) for data collection with force feedback. The virtual and real robots are aligned, ensuring that all movements of the real robot are accurately projected onto the virtual counterpart.
• Figure 2: Two types of robot control interfaces with force feedback. The force can be perceived either through the vibration of the motion controller or the applied force from the real robot. These images are captured from the perspective of XR headsets.
• Figure 3: The four tasks used in the user study to evaluate the performance of each interface. The top row shows images captured from VR headsets, while the bottom row presents the corresponding tasks in simulation.
• Figure 4: Mean and standard error of completion time for different tasks and interfaces. The bars represent the mean completion time for each interface (MC, MCV, KT, KTF) across different tasks (Push, Pick, Assemble and Open). The thick black outlines indicate the standard error, showing the variability in task completion times.
• Figure 5: Boxplots for different subjective metrics across tasks and interfaces.