Decoding Psychological States Through Movement: Inferring Human Kinesic Functions with Application to Built Environments

TL;DR

This work tackles the challenge of measuring how built environments influence social capital by defining a function-level interaction vocabulary grounded in Ekman and Friesen’s kinesics taxonomy. It introduces DUET, a dyadic, privacy-preserving dataset with 12 interactions across five kinesic functions collected via four modalities in three locations, and an embedded kinesics recognition framework that learns from skeletal motion without handcrafted action-to-function mappings. Benchmark results reveal the limitations of monadic HAR in capturing dyadic, socially meaningful interactions, while the proposed transfer-learning pipeline demonstrates that latent spatiotemporal representations correlate with downstream kinesics classification (Pearson $\rho=0.91$, $p=2\times10^{-6}$). The work establishes DUET and the framework as foundations for scalable, theory-grounded measurement of interaction in built environments, enabling closed-loop, evidence-based design and evaluation of social infrastructure.

Abstract

Social infrastructure and other built environments are increasingly expected to support well-being and community resilience by enabling social interaction. Yet in civil and built-environment research, there is no consistent and privacy-preserving way to represent and measure socially meaningful interaction in these spaces, leaving studies to operationalize "interaction" differently across contexts and limiting practitioners' ability to evaluate whether design interventions are changing the forms of interaction that social capital theory predicts should matter. To address this field-level and methodological gap, we introduce the Dyadic User Engagement DataseT (DUET) dataset and an embedded kinesics recognition framework that operationalize Ekman and Friesen's kinesics taxonomy as a function-level interaction vocabulary aligned with social capital-relevant behaviors (e.g., reciprocity and attention coordination). DUET captures 12 dyadic interactions spanning all five kinesic functions-emblems, illustrators, affect displays, adaptors, and regulators-across four sensing modalities and three built-environment contexts, enabling privacy-preserving analysis of communicative intent through movement. Benchmarking six open-source, state-of-the-art human activity recognition models quantifies the difficulty of communicative-function recognition on DUET and highlights the limitations of ubiquitous monadic, action-level recognition when extended to dyadic, socially grounded interaction measurement. Building on DUET, our recognition framework infers communicative function directly from privacy-preserving skeletal motion without handcrafted action-to-function dictionaries; using a transfer-learning architecture, it reveals structured clustering of kinesic functions and a strong association between representation quality and classification performance while generalizing across subjects and contexts.

Figures (10)

• Figure 1: (a) Existing literature provides three separate sets of evidence: (i) studies showing that social infrastructure shapes social interactions (Social Infrastructure $\rightarrow$ yellow oval), (ii) studies showing that social interactions contribute to social capital (orange oval $\rightarrow$ Social Capital), and (iii) studies correlating social infrastructure directly with social capital (Social Infrastructure $\rightarrow$ Social Capital). However, no existing work connects these components through a shared, measurable representation of the interaction layer, outlined in red. (b) This paper addresses that gap by introducing a quantitative, kinesics-based approach for observing the interaction mechanisms that mediate the effects of spatial design on social capital, enabling a systematic pathway from social infrastructure to interactional function to social capital.
• Figure 2: Labeling scheme for the 32 skeleton joints extracted using the Azure Kinect noauthor_azure_nodate.
• Figure 3: Sample data from 12 interactions. The modalities are (top row to bottom row): RGB, IR, depth, and 3D skeleton joints. The 12 interactions are (left to right): "waving in," "thumbs up," "waving," "pointing," "showing measurements," "nodding," "drawing circles in the air," "holding one's palms out," "twirling or scratching hair," "laughing," "arm crossing," and "hugging."
• Figure 4: (a) The sensing module configuration and (b) bird's-eye view of the testbed remain consistent across locations, with subjects confined to a rectangular area.
• Figure 5: Testing locations: (a) open indoor area, (b) confined indoor space, and (c) outdoor area.