Initial Findings on Sensor based Open Vocabulary Activity Recognition via Text Embedding Inversion

TL;DR

Open Vocabulary HAR (OV-HAR) addresses the limitation of traditional HAR classifiers by enabling recognition of unseen activities through open vocabulary text descriptions. The method translates activities into natural-language sequences of atomic motions, encodes them as a fixed $768$-dimensional embedding via a two-stage text pipeline, and learns a regression from sensor data to this embedding, with a pretrained embedding inversion to recover text and prompt-based mapping back to discrete classes. Evaluations across vision pose, IMU, and pressure-sensor data show that OV-HAR generalizes to unseen activities and modalities while avoiding the computational overhead of autoregressive LLMs, offering a practical balance between performance and efficiency. This approach paves the way for interpretable, edge-friendly open-vocabulary HAR with potential for flexible downstream prompting and multi-modal fusion.

Abstract

Conventional human activity recognition (HAR) relies on classifiers trained to predict discrete activity classes, inherently limiting recognition to activities explicitly present in the training set. Such classifiers would invariably fail, putting zero likelihood, when encountering unseen activities. We propose Open Vocabulary HAR (OV-HAR), a framework that overcomes this limitation by first converting each activity into natural language and breaking it into a sequence of elementary motions. This descriptive text is then encoded into a fixed-size embedding. The model is trained to regress this embedding, which is subsequently decoded back into natural language using a pre-trained embedding inversion model. Unlike other works that rely on auto-regressive large language models (LLMs) at their core, OV-HAR achieves open vocabulary recognition without the computational overhead of such models. The generated text can be transformed into a single activity class using LLM prompt engineering. We have evaluated our approach on different modalities, including vision (pose), IMU, and pressure sensors, demonstrating robust generalization across unseen activities and modalities, offering a fundamentally different paradigm from contemporary classifiers.

Figures (1)

• Figure 1: Comparison of the OV-HAR framework with contemporary classifiers. Unlike traditional classifiers, which are restricted to predicting only the classes present in their training set, OV-HAR can accurately recognize activities beyond its training set without requiring few-shot learning or fine-tuning.