Integrating Open-World Shared Control in Immersive Avatars

TL;DR

Teleoperation of immersive avatar robots enables remote manipulation but struggles with proficiency in open-world tasks. The authors propose a framework that combines open-world shared control with an immersive avatar interface, featuring an in-headset hierarchical pie menu, AR affordances, and a structured intent predictor to offer direct, shared, and autonomous control. Key contributions include affordance-based assistive actions, a predictive menu that reduces choices to a top set, and an open-world predictor trained on $150$ expert demonstrations, validated by a $N=19$ novice human-subject study showing improved success rates and shorter task times while maintaining presence. The results demonstrate that immersive teleoperation can benefit from open-world assistance without sacrificing immersion, suggesting scalable applicability to hazardous or complex environments.

Abstract

Teleoperated avatar robots allow people to transport their manipulation skills to environments that may be difficult or dangerous to work in. Current systems are able to give operators direct control of many components of the robot to immerse them in the remote environment, but operators still struggle to complete tasks as competently as they could in person. We present a framework for incorporating open-world shared control into avatar robots to combine the benefits of direct and shared control. This framework preserves the fluency of our avatar interface by minimizing obstructions to the operator's view and using the same interface for direct, shared, and fully autonomous control. In a human subjects study (N=19), we find that operators using this framework complete a range of tasks significantly more quickly and reliably than those that do not.

Figures (5)

• Figure 1: An operator uses the Avatar robot to unscrew a jar using the immersive interface. The predictive menu suggests possible assistive actions and shows corresponding affordances as augmented-reality objects (purple circle overlaying the jar lid). [Best viewed in color.]
• Figure 2: System diagram showing how different interface elements control the robot. Operators use their own head and hand to control the robot's head and hand, and use a button on their hand controller to interact with the assistive menu. The Perception Module detects affordances in the environment to display possible assistive actions to the operator. [Best viewed in color.]
• Figure 3: Flow diagram showing how different menus are accessed. Depending on which interface type is being used, the B button will show the operator different interfaces: in manual mode, this button will directly show the manual menu, while in predictive mode, it will show the predictive menu. In the predictive menu shown here, each teleop icon gives the operator the option to choose a different set of constraints. Orange emphasis is added to highlight certain icons, and is not present in the actual menu. [Best viewed in color.]
• Figure 4: 2D illustration of the snap_to_plane and snap_to_circle actions. Both align TRINA's gripper with the normal of the selected affordance, but snap_to_circle centers the gripper on the circle while snap_to_plane only moves it closer to the plane. Here we used $d_s = 0.15$ m. [Best viewed in color.]
• Figure 5: The three testing tasks. Target objects are highlighted with orange circles. [Best viewed in color.]