Interactive Robot Programming for Surface Finishing via Task-Centric Mixed Reality Interfaces

TL;DR

The paper tackles the barrier of programming collaborative robots for surface finishing in small-batch, high-variability scenarios by introducing a task-centric mixed reality interface and a contact-point-guided surface segmentation algorithm. It integrates user input with a robust RANSAC-based fitting process over surface primitives, generating a processing surface and a grid-based trajectory, with continuous visual feedback to align user intent with the learned model. Through two user studies, it demonstrates that a tablet-based GUI with projector augmentation (T-T) minimizes workload, enhances usability, and enables novices to rapidly program effective sanding tasks, outperforming HMD-based and non-feedback designs. This work advances practical MR-assisted robot programming by reducing setup complexity, improving model comprehension, and delivering scalable, configurable surface-finishing workflows for SMEs.

Abstract

Lengthy setup processes that require robotics expertise remain a major barrier to deploying robots for tasks involving high product variability and small batch sizes. As a result, collaborative robots, despite their advanced sensing and control capabilities, are rarely used for surface finishing in small-scale craft and manufacturing settings. To address this gap, we propose a novel robot programming approach that enables non-experts to intuitively program robots through interactive, task-focused workflows. For that, we developed a new surface segmentation algorithm that incorporates human input to identify and refine workpiece regions for processing. Throughout the programming process, users receive continuous visual feedback on the robot's learned model, enabling them to iteratively refine the segmentation result. Based on the segmented surface model, a robot trajectory is generated to cover the desired processing area. We evaluated multiple interaction designs across two comprehensive user studies to derive an optimal interface that significantly reduces user workload, improves usability and enables effective task programming even for users with limited practical experience.

Figures (9)

• Figure 1: Our proposed human-robot interface for programming surface finishing tasks. The task-centric approach allows users to focus on the task to perform on the workpiece, instead of programming the robot's motions step-by-step.
• Figure 2: The iterative bidirectional interaction between user and robot illustrated with three different objects.
• Figure 3: Setup of the box sanding task, where the upper plane of the box (marked in green) should be sanded by the robot, without processing the corner protectors highlighted in blue. As shown on the right, the participants can interact with the system via a head-mounted MR device and the robot in gravity compensation mode.
• Figure 4: The five user interface designs compared in the box sanding study. In R-X, no feedback is provided and the participants can not edit the segmentation result after the demonstration. The other interfaces highlight object inliers in blue and contact points in pink. In the edit phase, areas to exclude from processing can be selected through the HMD.
• Figure 5: The overall model understanding score and the NASA TLX mental demand for the interfaces of the box sanding user study. Error bars indicate 95% confidence intervals, statistical significance level $\bm{*}: p < 0.05$, $\bm{**}: p < 0.001$.