Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Multimodal Data Collection Framework for Dialogue-Driven Assistive Robotics to Clarify Ambiguities: A Wizard-of-Oz Pilot Study

Guangping Liu, Nicholas Hawkins, Billy Madden, Tipu Sultan, Flavio Esposito, Madi Babaiasl

TL;DR

The paper addresses the lack of multimodal, dialogue-grounded datasets for assistive mobile manipulation by introducing a two-room Wizard-of-Oz data collection framework that records five synchronized modalities during five daily tasks. It extends OpenTeach to support coordinated wheelchair and WMRA control, enabling natural, multi-turn dialogue while a human teleoperator simulates autonomy. A pilot study collected 53 trials from five participants, validated data quality through motion-smoothness metrics and participant feedback, and analyzed dialogue ambiguity types across tasks. The work provides a high-fidelity dataset and an extensible framework to benchmark and learn ambiguity-aware autonomous control for assistive devices, with the potential to scale to larger datasets and more complex tasks.

Abstract

Integrated control of wheelchairs and wheelchair-mounted robotic arms (WMRAs) has strong potential to increase independence for users with severe motor limitations, yet existing interfaces often lack the flexibility needed for intuitive assistive interaction. Although data-driven AI methods show promise, progress is limited by the lack of multimodal datasets that capture natural Human-Robot Interaction (HRI), particularly conversational ambiguity in dialogue-driven control. To address this gap, we propose a multimodal data collection framework that employs a dialogue-based interaction protocol and a two-room Wizard-of-Oz (WoZ) setup to simulate robot autonomy while eliciting natural user behavior. The framework records five synchronized modalities: RGB-D video, conversational audio, inertial measurement unit (IMU) signals, end-effector Cartesian pose, and whole-body joint states across five assistive tasks. Using this framework, we collected a pilot dataset of 53 trials from five participants and validated its quality through motion smoothness analysis and user feedback. The results show that the framework effectively captures diverse ambiguity types and supports natural dialogue-driven interaction, demonstrating its suitability for scaling to a larger dataset for learning, benchmarking, and evaluation of ambiguity-aware assistive control.

A Multimodal Data Collection Framework for Dialogue-Driven Assistive Robotics to Clarify Ambiguities: A Wizard-of-Oz Pilot Study

TL;DR

The paper addresses the lack of multimodal, dialogue-grounded datasets for assistive mobile manipulation by introducing a two-room Wizard-of-Oz data collection framework that records five synchronized modalities during five daily tasks. It extends OpenTeach to support coordinated wheelchair and WMRA control, enabling natural, multi-turn dialogue while a human teleoperator simulates autonomy. A pilot study collected 53 trials from five participants, validated data quality through motion-smoothness metrics and participant feedback, and analyzed dialogue ambiguity types across tasks. The work provides a high-fidelity dataset and an extensible framework to benchmark and learn ambiguity-aware autonomous control for assistive devices, with the potential to scale to larger datasets and more complex tasks.

Abstract

Integrated control of wheelchairs and wheelchair-mounted robotic arms (WMRAs) has strong potential to increase independence for users with severe motor limitations, yet existing interfaces often lack the flexibility needed for intuitive assistive interaction. Although data-driven AI methods show promise, progress is limited by the lack of multimodal datasets that capture natural Human-Robot Interaction (HRI), particularly conversational ambiguity in dialogue-driven control. To address this gap, we propose a multimodal data collection framework that employs a dialogue-based interaction protocol and a two-room Wizard-of-Oz (WoZ) setup to simulate robot autonomy while eliciting natural user behavior. The framework records five synchronized modalities: RGB-D video, conversational audio, inertial measurement unit (IMU) signals, end-effector Cartesian pose, and whole-body joint states across five assistive tasks. Using this framework, we collected a pilot dataset of 53 trials from five participants and validated its quality through motion smoothness analysis and user feedback. The results show that the framework effectively captures diverse ambiguity types and supports natural dialogue-driven interaction, demonstrating its suitability for scaling to a larger dataset for learning, benchmarking, and evaluation of ambiguity-aware assistive control.
Paper Structure (32 sections, 4 equations, 5 figures, 3 tables)

This paper contains 32 sections, 4 equations, 5 figures, 3 tables.

Table of Contents

  1. Abstract
  2. Introduction
  3. Related Work
  4. Multimodal Datasets for Assistive Robotics
  5. Natural Language and Dialogue in HRI
  6. Teleoperation
  7. Integrated Robotic System
  8. Teleoperation and Data Collection Framework
  9. Wizard-of-Oz Protocol
  10. WoZ Scheme Setup
  11. Experimental Design
  12. Experiment Environment
  13. Task Design
  14. Participants
  15. Study Procedure
  16. ...and 17 more sections

Figures (5)

  • Figure 1: Overview of the Multimodal Assistive Data Collection Framework: Our assistive data collection framework comprises a Virtual Reality (VR) based teleoperation system for a wheelchair and robotic arm, and a real-time multimodal data recording pipeline. The experimental setup spans two physically separated spaces (Room A and Room B). Room A is arranged with five assistive tasks, including door opening, drawer opening, drinking, feeding, and cleaning. Room B houses the teleoperator, who follows a Wizard-of-Oz (WoZ) protocol riek2012wizard to simulate robot autonomy and elicit natural dialogue. The resulting dataset contains 53 trials with five synchronized modalities, including RGB-D video, human-robot conversational audio, Inertial Measurement Unit (IMU) measurements, and robot kinematics, which are end-effector (EE) poses and whole-body joint states.
  • Figure 2: WheelArm Assistive Platform Hardware Overview: (1) Kinova Gen3 arm (6-DoF) with Intel RealSense D415 camera; (2) WHILL Model CR2 wheelchair; (3) Luxonis OAK-D-W camera and IMU; (4) edge device for teleoperation and data collection; (5) back shelf; (6) robotic arm adapter; (7) precharge switch; (8) precharge module; (9) main contactor; (10, 11) circuit breakers (15 A, 20 A); (12) two 12 V lithium batteries.
  • Figure 3: Teleoperation and data collection framework. Strikethrough elements indicate components from the initial software version that were updated or extended in this work.
  • Figure 4: Quantitative Analysis: (a-e) Task Performance Analysis: (a) is the task distribution in the pilot dataset, (b) is total completion time (mean $\pm$ SD) by task, (c) is end-effector (EE) path length in meters (mean $\pm$ SD) across the five tasks, capturing both average motion extent and variability, (d) is wheelchair mean jerk (mean $\pm$ SD); nonzero jerk is primarily observed in the door-opening task, where the wheelchair is repositioned, while remaining smooth, and (e) is EE mean jerk (mean $\pm$ SD), indicating consistently low jerk and thus smooth, unjittered EE motion during data collection. (f-h) Dialogue Analysis: (f) is task distribution in each ambiguity, (h) is the dialogue-ambiguity distribution for each task, and (g) is utterance counts of each task. User Feedback Analysis: (i) Likert Response Distribution demonstrates how subjects like this dialogue-based interaction and an intelligent robot on assistive tasks.
  • Figure 5: Task Performance Qualitative Demonstration.