TxP: Reciprocal Generation of Ground Pressure Dynamics and Activity Descriptions for Improving Human Activity Recognition

TL;DR

TxP tackles data scarcity in pressure-based HAR by establishing a bidirectional Text×Pressure framework that translates natural language into dynamic pressure maps and vice versa. Central to the approach is PressureRQVAE, which quantizes pressure dynamics into discrete tokens, enabling Text2Pressure to synthesize realistic pressure sequences from text and Pressure2Text to generate descriptive text from pressure data using an LLM backbone. Across multiple real-world HAR datasets, TxP delivers substantial improvements in macro F1 and related metrics, while also enabling scalable data augmentation and enhanced interpretability through atomic-action level descriptions. The framework demonstrates practical potential for wearables and cross-modal HAR tasks, with clear pathways to extend to additional sensor modalities and to mitigate biases introduced by large-language models.

Abstract

Sensor-based human activity recognition (HAR) has predominantly focused on Inertial Measurement Units and vision data, often overlooking the capabilities unique to pressure sensors, which capture subtle body dynamics and shifts in the center of mass. Despite their potential for postural and balance-based activities, pressure sensors remain underutilized in the HAR domain due to limited datasets. To bridge this gap, we propose to exploit generative foundation models with pressure-specific HAR techniques. Specifically, we present a bidirectional Text$\times$Pressure model that uses generative foundation models to interpret pressure data as natural language. TxP accomplishes two tasks: (1) Text2Pressure, converting activity text descriptions into pressure sequences, and (2) Pressure2Text, generating activity descriptions and classifications from dynamic pressure maps. Leveraging pre-trained models like CLIP and LLaMA 2 13B Chat, TxP is trained on our synthetic PressLang dataset, containing over 81,100 text-pressure pairs. Validated on real-world data for activities such as yoga and daily tasks, TxP provides novel approaches to data augmentation and classification grounded in atomic actions. This consequently improved HAR performance by up to 12.4\% in macro F1 score compared to the state-of-the-art, advancing pressure-based HAR with broader applications and deeper insights into human movement.

Figures (10)

• Figure 1: The TxP framework consists of two key components: (i) Text2Pressure, which generates dynamic pressure sequences from textual descriptions similar to CLIP radford2021learning, and (ii) Pressure2Text, which creates activity descriptions from dynamic pressure sequences similar to PaLM-E driess2023palm. At its core is PressureRQVAE, which creates a mapping between raw pressure dynamics and a Pressure codebook that contains pressure embeddings represented in a discrete vector space.
• Figure 2: We used PressureRQVAE to quantize continuous dynamic length pressure time series data into discrete NLP-like tokens. The figure showcases one example of seven-second continuous pressure dynamics used to generate a discrete vector sequence of five tokens. Each token represents two seconds of pressure sequence data, the fourth token has only one second of pressure data and another second of null data (all pressure points have a null value), and finally, the fifth end token $V_x$ represents the end of the pressure sequence.
• Figure 3: Text2Pressure uses a frozen CLIP text encoder to convert text into vector embeddings, which are then given as input to a trainable autoregressive transformer to generate T tokens up to $V_x$. The generated tokens are fed into a pre-trained PressureRQVAE decoder to produce the pressure dynamics.
• Figure 4: Pressure2Text utilizes a pre-trained PressureRQVAE encoder to transform pressure dynamics into NLP-like tokens. These tokens pass through a trainable projection head, aligning with text tokens. Both the aligned pressure tokens and text tokens are then fed into a frozen LLaMA 2 13B Chat model.
• Figure 5: We showcase how both components of TxP can be utilized to create a better HAR system where LLaMA 2, along with Text2Pressure, is used to create a synthetic dataset with variations from the original activity labels while Pressure2Text along with prompt engineering, is utilized to generate an activity label from a dynamic pressure map sequence. We evaluated improvement on HAR by using our data augmentation method (Text2Pressure), grounded classification with atomic action (Pressure2Text), and both together as compared to the contemporary HAR system that uses a traditional classifier (our baseline and other SOTA model on different dataset) to predict the activity label from a fixed window of pressure sensor sequence.