Physiologically-Informed Predictability of a Teammate's Future Actions Forecasts Team Performance

TL;DR

The paper addresses how multi-human team performance relates to behavioral and physiological markers in collaborative settings. It introduces a triadic VR task (ADCT) and a cross-modal, transformer-based predictability model that forecasts one teammate's future actions using others' data, contrasting this with inter-subject synchrony analyses. The key finding is that the predictability biomarker aligns positively with team performance (e.g., $eta=3.20$, $P<0.001$), while physiological synchrony shows limited predictive power; speech coordination enhances performance whereas over-coordination in controller actions may hinder it. This work provides a quantitative framework for understanding and optimizing multi-human teaming and points toward avenues for human-AI collaboration in complex, coordinated tasks.

Abstract

In collaborative environments, a deep understanding of multi-human teaming dynamics is essential for optimizing performance. However, the relationship between individuals' behavioral and physiological markers and their combined influence on overall team performance remains poorly understood. To explore this, we designed a triadic human collaborative sensorimotor task in virtual reality (VR) and introduced a novel predictability metric to examine team dynamics and performance. Our findings reveal a strong connection between team performance and the predictability of a team member's future actions based on other team members' behavioral and physiological data. Contrary to conventional wisdom that high-performing teams are highly synchronized, our results suggest that physiological and behavioral synchronizations among team members have a limited correlation with team performance. These insights provide a new quantitative framework for understanding multi-human teaming, paving the way for deeper insights into team dynamics and performance.

Figures (4)

• Figure 1: ADCT environment and team performance. a, An illustration of ADCT virtual environment. The team's goal is to control the spacecraft, passing all red rings and arrive back to Earth within the specified time limit. b, The view of three co-pilots with respect to a ring obstacle and the degree of freedom controlled by each role. The three co-pilots are YawPilot, PitchPilot, and ThrustPilot. Each participant was equipped with a VR headset, a microphone, a remote controller, and an EEG headset. c, Illustration of data modalities collected from all co-pilots. The red bars on the spacecraft's horizontal and vertical trajectories represent the relative location of ring obstacles. The uppermost section illustrates the cross-section of a spacecraft's position with respect to a ring. d-e, Team performance across three experimental sessions. The performance is measured by the number of rings passed. Each dot indicates one team (N = 17). Bars indicate the average across teams. d, Total number of rings passed by each team in each session. e, Averaged the number of rings passed by each team in each trial in three sessions. Asterisks indicate statistically significant differences, defined as ns, not significant, $***P < 0.001.$
• Figure 2: Subjective rating and multi-modal synchrony. a-f, Subjective ratings and synchrony among team members based on different physiological or behavioral data modalities across experimental sessions. Each dot represents one team, and the bars show the average across teams. a, Helpfulness rating of team members (N=17). b, Familiarity rating of team members (N=17). c, Pupil size synchrony among team members (N=14). d, EEG synchrony among team members (N=9). e, Remote controller action synchrony among team members (N=17). f, Speech event synchrony among team members (N=7). g, Multi-modal synchrony and its correlation with team performance. Blue arrows indicate negative correlations, while red arrows indicate positive correlations. Asterisks indicate statistically significant differences, defined as ns, not significant, $\boldsymbol{\cdot} P<0.1, *P < 0.05, ***P < 0.001.$
• Figure 3: Predictability of each team member's actions as a biomarker of team performance. a, An illustration of a single epoch of the multi-modal data. Each epoch is relative to a ring, and we divided each epoch into input and output for the predictive model. The predicted action of an individual is based on a generative model that uses the behavioral and physiological data of the other two co-pilots. Predictability is evaluated by computing the correlation of the true action of a co-pilot with the model-predicted action. b, Multi-head attention modal structure. The cross-modal attention layers take the spacecraft trajectories and physiological or behavioral data. c Team predictability across three experimental sessions. Each dot represents one team, and the bars show the average across teams ($n=10$). ns, not significant. d, Correlation between team performance and predictability. The red arrow indicates positive correlations. Asterisks indicate statistically significant differences, defined as $***P < 0.001.$
• Figure 4: Overview of the correlation between predictability and team performance. All correlations are the GLMM results in accounting for session-level differences. The predictability biomarker is computed based on multi-modal physiological and behavioral data. Each arrow originates from the independent variable and points towards the dependent variable. The blue arrow indicates negative correlations, while the red arrow indicates positive correlations. Dashed lines indicate insignificant correlations. $\boldsymbol{\cdot}~P < 0.1, * P < 0.05, *** P < 0.001$, n = 10.