Vision-Based Multimodal Interfaces: A Survey and Taxonomy for Enhanced Context-Aware System Design

TL;DR

This survey addresses a gap in VMIs by adopting a data-modality-driven lens, prioritizing the visual modality for context understanding while integrating non-visual data. It introduces a Macro-Micro-Macro (3M) system design framework to guide from holistic context to modality-level details and back to synthesis, and provides a PRISMA-based literature analysis of 109 papers. The authors deliver a structured taxonomy (context factors, context categories, input modalities, data integration stages, processing, and evaluation) plus design considerations and open challenges, intended as a practical blueprint for practitioners. The work’s practical impact lies in offering actionable guidance, an interactive knowledge tool, and a pathway for scalable, context-aware HCI across domains such as AR/VR, healthcare, education, and accessibility.

Abstract

The recent surge in artificial intelligence, particularly in multimodal processing technology, has advanced human-computer interaction, by altering how intelligent systems perceive, understand, and respond to contextual information (i.e., context awareness). Despite such advancements, there is a significant gap in comprehensive reviews examining these advances, especially from a multimodal data perspective, which is crucial for refining system design. This paper addresses a key aspect of this gap by conducting a systematic survey of data modality-driven Vision-based Multimodal Interfaces (VMIs). VMIs are essential for integrating multimodal data, enabling more precise interpretation of user intentions and complex interactions across physical and digital environments. Unlike previous task- or scenario-driven surveys, this study highlights the critical role of the visual modality in processing contextual information and facilitating multimodal interaction. Adopting a design framework moving from the whole to the details and back, it classifies VMIs across dimensions, providing insights for developing effective, context-aware systems.

Figures (7)

• Figure 1: The publication growth trend for vision-based multimodal interface and context awareness in the ACM Digital Library (details of the search keywords are provided in Appendix \ref{['select_literature']}).
• Figure 2: Examples of context source factors in VMIs with descriptions and citations (illustrative references (a): Bethge2021VEmotionUD, (b): Rawat2017ClickSmartAC, (c): Saad2023HotFootFU, (d): Chen2023PaperToPlaceTI, (e): Wang2024GVOILAGI, (f): Dogan2021SensiCutML, (g): Chen2023PaperToPlaceTI, (h): Lee2024GazePointARAC).
• Figure 3: Examples of context categories in VMIs with descriptions and citations (illustrative references (a): zhou2022gesture, (b): Yeo2017SpeCamSS, (c): Liang2021AuthTrackEA, (d): Lee2024GazePointARAC).
• Figure 4: Examples of application domains for VMIs (illustrative references (a): hu2023microcam, (b): Liang2021AuthTrackEA, (c): Saad2023HotFootFU, (d): kong2021eyemu, (e): Hautasaari2024EmoScribeCA, (f): Liao2023GPT4EM, (g): Wen2024AdaptiveVoiceCA, (h): Malawade2022HydraFusionCS, (i): Doudou2019DriverDM, (j): Tsai2024GazeNoterCA, (k): Fan2024ContextCamBC, (l): hoang2024artvista, (m): Wentzel2024SwitchSpaceUC, (n): de2024llmr, (o): bokaris2019light, (p): Lim2024ExploringCM, (q): hu2024exploring, (r): Englhardt2023FromCT, (s): Zhu2023IntegratingGA, (t): zhou2022gesture, (u): Zhang2024MathemythsLL, (v): Su2024RASSARRA, (w): Wang2024GazePromptEL, (x): Wang2024GazePromptEL, (y): zargham2024know, (z): somarathna2023exploring, (&): Suzuki2019AnOM).
• Figure 5: Examples of design considerations and key challenges for VMIs (illustrative references (a): Khan2021PALWA, (b): lee2018anthropometric, (c): de2024llmr, (d): Wen2024AdaptiveVoiceCA, (e): perera2013dynamic, (f): spanos2012sensorstream, (g): Liao2023GPT4EM, (h): Koch2023LeveragingDV, (i): pahde2021multimodal, (j): Matsuda2018EmoTourME, (k): McDuff2019AME, (l): bouchard2020ear.