MENA: Multimodal Epistemic Network Analysis for Visualizing Competencies and Emotions

TL;DR

This work tackles the challenge of understanding how caregiver competencies and emotions unfold during AR-based geriatric simulations. It introduces Multimodal Epistemic Network Analysis (MENA), which fuses an Emotional State Classifier with ENA to map co-occurring nursing-competency codes and emotions across audio, 3D pose, text, and a ConceptNet knowledge-graph, using $C_p$ adjacency structures and $SVD_1$/$SVD_2$ axes. In a within-subject pilot with $n=20$, the study shows that VGP awareness fosters more empathetic, person-centered care, as reflected in denser connections and a significant $t$-test on ENA scores ($t(55.83)=2.93$, $p<0.01$, Cohen’s $d=0.93$). The results indicate MENA as a scalable visualization framework for dynamic interpersonal training with broad applicability beyond geriatric care to other domains.

Abstract

The need to improve geriatric care quality presents a challenge that requires insights from stakeholders. While simulated trainings can boost competencies, extracting meaningful insights from these practices to enhance simulation effectiveness remains a challenge. In this study, we introduce Multimodal Epistemic Network Analysis (MENA), a novel framework for analyzing caregiver attitudes and emotions in an Augmented Reality setting and exploring how the awareness of a virtual geriatric patient (VGP) impacts these aspects. MENA enhances the capabilities of Epistemic Network Analysis by detecting positive emotions, enabling visualization and analysis of complex relationships between caregiving competencies and emotions in dynamic caregiving practices. The framework provides visual representations that demonstrate how participants provided more supportive care and engaged more effectively in person-centered caregiving with aware VGP. This method could be applicable in any setting that depends on dynamic interpersonal interactions, as it visualizes connections between key elements using network graphs and enables the direct comparison of multiple networks, thereby broadening its implications across various fields.

Figures (6)

• Figure 1: Augmented Reality setup and environment for caregiving practice: (a) the 3D representation of a virtual geriatric patient in a nursing home setting. (b) interaction between a participant and the VGP.
• Figure 2: Excerpt of coded data with a schematic design of the Multimodal Matrix used in MENA.
• Figure 3: Adjacency matrices, displaying the co-occurrence of codes, for the two stanzas presented in Table \ref{['fig:Coded_Data']}. The adjacency matrix has zero values in diagonal cells, and is symmetric.
• Figure 4: The cumulative adjacency matrix of two stanzas for a participant, identified by Participant1, Which sums the adjacency matrices presented in Fig \ref{['fig:Adjacency matrices']}.
• Figure 5: MENA representations of participant behavioral dynamics during interactions with unaware (red) and aware (blue) VGP. PVD stands for Patient Vulnerability Disclosure. In the model, the size of each node corresponds to the frequency with which the code appears, while the thickness of the lines between nodes indicates the strength of the connections. In general, participants demonstrated denser connections on the left side of the X-axis in the aware condition, exhibiting more positive emotions around Person-Centered Care and in response to Patient Vulnerability Disclosure. Conversely, in the unaware condition, participants showed stronger connections on the right side of the X-axis, relying on established protocols.