Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Augmented Body Communicator: Enhancing daily body expression for people with upper limb limitations through LLM and a robotic arm

Songchen Zhou, Mark Armstrong, Giulia Barbareschi, Toshihiro Ajioka, Zheng Hu, Ryoichi Ando, Kentaro Yoshifuji, Masatane Muto, Kouta Minamizawa

TL;DR

This paper tackles the challenge of enabling nonverbal social expression for individuals with upper-limb mobility limitations. It introduces the Augmented Body Communicator (ABC), a system that couples a 7-DoF robotic arm with kinetic memory to express gestures, a camera-based LLM (gpt-4o Vision) to interpret partner cues, and a web-based control interface for both users and assistants. Through a two-study program, it demonstrates high alignment between LLM recommendations and human preferences (Study 1) and reports high usability and reduced workload for disabled users in real-world use (Study 2), while also noting latency and initiation constraints. The work advances inclusive HCI by proposing a pathway toward holistic ACCs that blend verbal and nonverbal communication, informing future design of AI-enabled assistive devices for daily-life social interaction.

Abstract

Individuals with upper limb movement limitations face challenges in interacting with others. Although robotic arms are currently used primarily for functional tasks, there is considerable potential to explore ways to enhance users' body language capabilities during social interactions. This paper introduces an Augmented Body Communicator system that integrates robotic arms and a large language model. Through the incorporation of kinetic memory, disabled users and their supporters can collaboratively design actions for the robot arm. The LLM system then provides suggestions on the most suitable action based on contextual cues during interactions. The system underwent thorough user testing with six participants who have conditions affecting upper limb mobility. Results indicate that the system improves users' ability to express themselves. Based on our findings, we offer recommendations for developing robotic arms that support disabled individuals with body language capabilities and functional tasks.

Augmented Body Communicator: Enhancing daily body expression for people with upper limb limitations through LLM and a robotic arm

TL;DR

This paper tackles the challenge of enabling nonverbal social expression for individuals with upper-limb mobility limitations. It introduces the Augmented Body Communicator (ABC), a system that couples a 7-DoF robotic arm with kinetic memory to express gestures, a camera-based LLM (gpt-4o Vision) to interpret partner cues, and a web-based control interface for both users and assistants. Through a two-study program, it demonstrates high alignment between LLM recommendations and human preferences (Study 1) and reports high usability and reduced workload for disabled users in real-world use (Study 2), while also noting latency and initiation constraints. The work advances inclusive HCI by proposing a pathway toward holistic ACCs that blend verbal and nonverbal communication, informing future design of AI-enabled assistive devices for daily-life social interaction.

Abstract

Individuals with upper limb movement limitations face challenges in interacting with others. Although robotic arms are currently used primarily for functional tasks, there is considerable potential to explore ways to enhance users' body language capabilities during social interactions. This paper introduces an Augmented Body Communicator system that integrates robotic arms and a large language model. Through the incorporation of kinetic memory, disabled users and their supporters can collaboratively design actions for the robot arm. The LLM system then provides suggestions on the most suitable action based on contextual cues during interactions. The system underwent thorough user testing with six participants who have conditions affecting upper limb mobility. Results indicate that the system improves users' ability to express themselves. Based on our findings, we offer recommendations for developing robotic arms that support disabled individuals with body language capabilities and functional tasks.
Paper Structure (23 sections, 5 figures, 1 table)

This paper contains 23 sections, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Augmentative and Alternative Communication
  4. Robot for Body Expression
  5. AI and Large Language Models for Inclusive Verbal and Nonverbal Communication
  6. Design of an Augmented Body Communicator system
  7. Exploratory Investigation and Design Concept
  8. System Implementation
  9. Physical Apparatus
  10. Control System
  11. Study 1: Accuracy testing of AI LLM system
  12. Participants and Procedure
  13. Findings
  14. Study 2: Evaluating the ABC system with Users with Limited Upper Limb Mobility
  15. Participants and Procedure
  16. ...and 8 more sections

Figures (5)

  • Figure 1: System Setup
  • Figure 2: System Workflow
  • Figure 3: Assistant and User Interfaces
  • Figure 4: Human Prefer Movement Rank Scores with LLM Indicated
  • Figure 5: ABC In-depth User test