Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Vision Beyond Boundaries: An Initial Design Space of Domain-specific Large Vision Models in Human-robot Interaction

Yuchong Zhang, Yong Ma, Danica Kragic

TL;DR

This work introduces an initial design space that incorporates domain-specific LVMs, chosen for their superior performance over normal models, and explores the process of ideation and potential application scenarios, envisioning this design space as a foundational guideline for future HRI system design.

Abstract

The emergence of large vision models (LVMs) is following in the footsteps of the recent prosperity of Large Language Models (LLMs) in following years. However, there's a noticeable gap in structured research applying LVMs to human-robot interaction (HRI), despite extensive evidence supporting the efficacy of vision models in enhancing interactions between humans and robots. Recognizing the vast and anticipated potential, we introduce an initial design space that incorporates domain-specific LVMs, chosen for their superior performance over normal models. We delve into three primary dimensions: HRI contexts, vision-based tasks, and specific domains. The empirical evaluation was implemented among 15 experts across six evaluated metrics, showcasing the primary efficacy in relevant decision-making scenarios. We explore the process of ideation and potential application scenarios, envisioning this design space as a foundational guideline for future HRI system design, emphasizing accurate domain alignment and model selection.

Vision Beyond Boundaries: An Initial Design Space of Domain-specific Large Vision Models in Human-robot Interaction

TL;DR

This work introduces an initial design space that incorporates domain-specific LVMs, chosen for their superior performance over normal models, and explores the process of ideation and potential application scenarios, envisioning this design space as a foundational guideline for future HRI system design.

Abstract

The emergence of large vision models (LVMs) is following in the footsteps of the recent prosperity of Large Language Models (LLMs) in following years. However, there's a noticeable gap in structured research applying LVMs to human-robot interaction (HRI), despite extensive evidence supporting the efficacy of vision models in enhancing interactions between humans and robots. Recognizing the vast and anticipated potential, we introduce an initial design space that incorporates domain-specific LVMs, chosen for their superior performance over normal models. We delve into three primary dimensions: HRI contexts, vision-based tasks, and specific domains. The empirical evaluation was implemented among 15 experts across six evaluated metrics, showcasing the primary efficacy in relevant decision-making scenarios. We explore the process of ideation and potential application scenarios, envisioning this design space as a foundational guideline for future HRI system design, emphasizing accurate domain alignment and model selection.
Paper Structure (14 sections, 2 figures)

This paper contains 14 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Design Space with HRI
  4. LVMs in HRI: status
  5. The Design Space
  6. HRI Contexts
  7. Vision-based tasks
  8. Specific domains
  9. Evaluation
  10. Study Design
  11. Data Analysis
  12. Summary of Qualitative Feedback
  13. Discussion
  14. Conclusion

Figures (2)

  • Figure 1: Our proposed design space. The HRI contexts interplay with vision-based tasks tailored to specific domains. Corresponding LVMs deliver performance that meets the unique requirements of each case.
  • Figure 2: A: Demographics: academic background information of all participants. B: The outcomes of the five evaluated metrics, which assessed the three dimensions and the overall design space. **: $p<0.05$.