Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Understanding On-the-Fly End-User Robot Programming

Laura Stegner, Yuna Hwang, David Porfirio, Bilge Mutlu

TL;DR

The paper investigates end-user experience with on-the-fly robot end-user programming using the Tabula prototype, revealing how multimodal inputs and program synthesis influence user understanding, control, and workflow. Through a 21-participant study involving structured and open-ended scenarios, five themes emerge, highlighting reliance on program-step visualization, non-sequential input challenges, tool-like perceptions of robots, mixed reactions to automated synthesis, and the learning curve beyond basic functionality. The authors derive design implications to improve feedback, support more step-wise input, clarify robot autonomy, and offer adjustable automation levels, culminating in Tabula 2.0 design directions. These findings advance understanding of EUP tool usability and provide actionable guidelines to better align on-the-fly programming interfaces with diverse user needs and domains. The work contributes to democratizing robot use by outlining practical pathways to design end-user friendly, autonomous-leaning but controllable robotic programming tools.

Abstract

Novel end-user programming (EUP) tools enable on-the-fly (i.e., spontaneous, easy, and rapid) creation of interactions with robotic systems. These tools are expected to empower users in determining system behavior, although very little is understood about how end users perceive, experience, and use these systems. In this paper, we seek to address this gap by investigating end-user experience with on-the-fly robot EUP. We trained 21 end users to use an existing on-the-fly EUP tool, asked them to create robot interactions for four scenarios, and assessed their overall experience. Our findings provide insight into how these systems should be designed to better support end-user experience with on-the-fly EUP, focusing on user interaction with an automatic program synthesizer that resolves imprecise user input, the use of multimodal inputs to express user intent, and the general process of programming a robot.

Understanding On-the-Fly End-User Robot Programming

TL;DR

The paper investigates end-user experience with on-the-fly robot end-user programming using the Tabula prototype, revealing how multimodal inputs and program synthesis influence user understanding, control, and workflow. Through a 21-participant study involving structured and open-ended scenarios, five themes emerge, highlighting reliance on program-step visualization, non-sequential input challenges, tool-like perceptions of robots, mixed reactions to automated synthesis, and the learning curve beyond basic functionality. The authors derive design implications to improve feedback, support more step-wise input, clarify robot autonomy, and offer adjustable automation levels, culminating in Tabula 2.0 design directions. These findings advance understanding of EUP tool usability and provide actionable guidelines to better align on-the-fly programming interfaces with diverse user needs and domains. The work contributes to democratizing robot use by outlining practical pathways to design end-user friendly, autonomous-leaning but controllable robotic programming tools.

Abstract

Novel end-user programming (EUP) tools enable on-the-fly (i.e., spontaneous, easy, and rapid) creation of interactions with robotic systems. These tools are expected to empower users in determining system behavior, although very little is understood about how end users perceive, experience, and use these systems. In this paper, we seek to address this gap by investigating end-user experience with on-the-fly robot EUP. We trained 21 end users to use an existing on-the-fly EUP tool, asked them to create robot interactions for four scenarios, and assessed their overall experience. Our findings provide insight into how these systems should be designed to better support end-user experience with on-the-fly EUP, focusing on user interaction with an automatic program synthesizer that resolves imprecise user input, the use of multimodal inputs to express user intent, and the general process of programming a robot.
Paper Structure (55 sections, 3 figures, 1 table)

This paper contains 55 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Approaches to End-user Specification
  4. End-User Development and Programming
  5. EUP Tool Usage
  6. Method
  7. Participants
  8. Interface
  9. Procedure
  10. Tutorial & Training
  11. Structured Scenarios
  12. Open-Ended Scenario
  13. Interview & Questionnaires
  14. Measures and Analysis
  15. Findings
  16. ...and 40 more sections

Figures (3)

  • Figure 1: We investigate end-user experience with on-the-fly robot end-user programming using Tabula, a state-of-the-art open-source research prototype. Right: An experimenter using speech and touch input to program a robot to put toys away in a toy chest. Left: A visual representation of the generated program by a study participant (P5).
  • Figure 2: We used Tabula, a multimodal EUP tool that uses a combination of speech and sketching input to generate a robot program porfirio2023sketching, to study end-user experiences with programming robots on the fly. Left: First, users configure the environment, including placing any objects for the robot to interact with. Middle: Second, users create recordings by first providing a speech utterance to instruct the robot what to do and subsequently creating a sketch by drawing a path of points of interest that the robot should visit. Right: Finally, inputs are combined by the program synthesizer, and users can view the resulting programming steps.
  • Figure 3: An overview of the connection between the findings and resulting design implications and recommendations.