Vision-Language System using Open-Source LLMs for Gestures in Medical Interpreter Robots

TL;DR

A privacy-preserving vision-language framework for medical interpreter robots that detects specific speech acts (consent and instruction) and generates corresponding robotic gestures and significantly improves computational efficiency and outperforms the speech-gesture generation baseline in human-likeness while maintaining comparable appropriateness.

Abstract

Effective communication is vital in healthcare, especially across language barriers, where non-verbal cues and gestures are critical. This paper presents a privacy-preserving vision-language framework for medical interpreter robots that detects specific speech acts (consent and instruction) and generates corresponding robotic gestures. Built on locally deployed open-source models, the system utilizes a Large Language Model (LLM) with few-shot prompting for intent detection. We also introduce a novel dataset of clinical conversations annotated for speech acts and paired with gesture clips. Our identification module achieved 0.90 accuracy, 0.93 weighted precision, and a 0.91 weighted F1-Score. Our approach significantly improves computational efficiency and, in user studies, outperforms the speech-gesture generation baseline in human-likeness while maintaining comparable appropriateness.

Figures (6)

• Figure 1: Overview of our proposed vision-language system for gesturing medical interpreter robots.
• Figure 2: Pipeline for using an LLM to create the clinical conversation dataset.
• Figure 3: The on-device gesture sentence detection module.
• Figure 4: Conversation videos to robot motions workflow.
• Figure 5: Confusion matrix illustrating the performance of qwen3:8b.