Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reframing Conversational Design in HRI: Deliberate Design with AI Scaffolds

Shiye Cao, Jiwon Moon, Yifan Xu, Anqi Liu, Chien-Ming Huang

TL;DR

This paper introduces ACE, an open-source AI-Aided Conversation Engine that enables deliberate, grounded design of LLM-powered human-robot conversations. ACE adds a voice-based prompt scaffold to escape the blank-page problem, a transcript-annotation interface for granular feedback, and LLM-driven translation of feedback into concrete prompt refinements. Through two studies, ACE demonstrated higher prompt clarity and specificity (Study 1) and improved end-user interaction quality (Study 2) compared with a Baseline tool, suggesting ACE reduces design friction and yields more engaging robot conversations. The work highlights a path toward holistic, evidence-based design tools for HRI and acknowledges limitations such as sample size and in-lab scope, proposing future work on multi-modal design and long-term validation.

Abstract

Large language models (LLMs) have enabled conversational robots to move beyond constrained dialogue toward free-form interaction. However, without context-specific adaptation, generic LLM outputs can be ineffective or inappropriate. This adaptation is often attempted through prompt engineering, which is non-intuitive and tedious. Moreover, predominant design practice in HRI relies on impression-based, trial-and-error refinement without structured methods or tools, making the process inefficient and inconsistent. To address this, we present the AI-Aided Conversation Engine (ACE), a system that supports the deliberate design of human-robot conversations. ACE contributes three key innovations: 1) an LLM-powered voice agent that scaffolds initial prompt creation to overcome the "blank page problem," 2) an annotation interface that enables the collection of granular and grounded feedback on conversational transcripts, and 3) using LLMs to translate user feedback into prompt refinements. We evaluated ACE through two user studies, examining both designs' experience and end users' interactions with robots designed using ACE. Results show that ACE facilitates the creation of robot behavior prompts with greater clarity and specificity, and that the prompts generated with ACE lead to higher-quality human-robot conversational interactions.

Reframing Conversational Design in HRI: Deliberate Design with AI Scaffolds

TL;DR

This paper introduces ACE, an open-source AI-Aided Conversation Engine that enables deliberate, grounded design of LLM-powered human-robot conversations. ACE adds a voice-based prompt scaffold to escape the blank-page problem, a transcript-annotation interface for granular feedback, and LLM-driven translation of feedback into concrete prompt refinements. Through two studies, ACE demonstrated higher prompt clarity and specificity (Study 1) and improved end-user interaction quality (Study 2) compared with a Baseline tool, suggesting ACE reduces design friction and yields more engaging robot conversations. The work highlights a path toward holistic, evidence-based design tools for HRI and acknowledges limitations such as sample size and in-lab scope, proposing future work on multi-modal design and long-term validation.

Abstract

Large language models (LLMs) have enabled conversational robots to move beyond constrained dialogue toward free-form interaction. However, without context-specific adaptation, generic LLM outputs can be ineffective or inappropriate. This adaptation is often attempted through prompt engineering, which is non-intuitive and tedious. Moreover, predominant design practice in HRI relies on impression-based, trial-and-error refinement without structured methods or tools, making the process inefficient and inconsistent. To address this, we present the AI-Aided Conversation Engine (ACE), a system that supports the deliberate design of human-robot conversations. ACE contributes three key innovations: 1) an LLM-powered voice agent that scaffolds initial prompt creation to overcome the "blank page problem," 2) an annotation interface that enables the collection of granular and grounded feedback on conversational transcripts, and 3) using LLMs to translate user feedback into prompt refinements. We evaluated ACE through two user studies, examining both designs' experience and end users' interactions with robots designed using ACE. Results show that ACE facilitates the creation of robot behavior prompts with greater clarity and specificity, and that the prompts generated with ACE lead to higher-quality human-robot conversational interactions.
Paper Structure (34 sections, 5 figures, 1 table)

This paper contains 34 sections, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Human-Robot Conversation Design
  4. Prompt Engineering
  5. System: AI-Aided Conversation Engine
  6. System Design
  7. Literature Review
  8. Interviews
  9. Formative Study
  10. Initial System Prototype
  11. Initial Prototype Testing
  12. Refined Prototype Testing
  13. Final System Overview
  14. Study 1: Evaluation from Designer Perspective
  15. Comparison to Baseline System
  16. ...and 19 more sections

Figures (5)

  • Figure 1: Main Design Page of ACE Interface. See supplementary materials for an enlarged view of the interface.
  • Figure 2: Effects of using Baseline and ACE in human-robot conversation design on prompt clarity (a), prompt specificity (b, c, d). Error bars shown represent standard error.
  • Figure 3: Effects of using Baseline and ACE on (a) participants' perceived satisfaction with the final prompt (b) SUS score. Error bars shown represent standard error.
  • Figure 4: Sample dialogue from Study 2 with robot using ACE prompt. Text in brackets indicates overlapping speech.
  • Figure 5: Effects of using Baseline and ACE prompts in human-robot conversation on a) goodness of interaction rating (bars represent standard error) and b) preferred design.