Sensor2Text: Enabling Natural Language Interactions for Daily Activity Tracking Using Wearable Sensors

TL;DR

Sensor2Text, a model proficient in tracking daily activities and engaging in conversations using wearable sensors, represents the first model capable of conversing about wearable sensor data, offering an innovative approach to daily activity tracking that addresses privacy and field of view limitations associated with current vision-based solutions.

Abstract

Visual Question-Answering, a technology that generates textual responses from an image and natural language question, has progressed significantly. Notably, it can aid in tracking and inquiring about daily activities, crucial in healthcare monitoring, especially for elderly patients or those with memory disabilities. However, video poses privacy concerns and has a limited field of view. This paper presents Sensor2Text, a model proficient in tracking daily activities and engaging in conversations using wearable sensors. The approach outlined here tackles several challenges, including low information density in wearable sensor data, insufficiency of single wearable sensors in human activities recognition, and model's limited capacity for Question-Answering and interactive conversations. To resolve these obstacles, transfer learning and student-teacher networks are utilized to leverage knowledge from visual-language models. Additionally, an encoder-decoder neural network model is devised to jointly process language and sensor data for conversational purposes. Furthermore, Large Language Models are also utilized to enable interactive capabilities. The model showcases the ability to identify human activities and engage in Q\&A dialogues using various wearable sensor modalities. It performs comparably to or better than existing visual-language models in both captioning and conversational tasks. To our knowledge, this represents the first model capable of conversing about wearable sensor data, offering an innovative approach to daily activity tracking that addresses privacy and field-of-view limitations associated with current vision-based solutions.

Figures (10)

• Figure 1: Visualization of body tracking data using wearable sensors. Different activities display distinct body positions, demonstrating the feasibility of using wearable sensors to identify daily activities.
• Figure 2: Architecture diagram of Sensor2Text.
• Figure 3: Example prompts and ground truths for the two aforementioned finetuning stages. After the text is tokenized into textual embeddings, the image or sensor embeddings outputted by the Querying Transformer are inserted at the indicated location, allowing the language model to process both the text and the image or sensor information when producing a response. For stage 1, using our hyperparameters, there are 64 sensor embeddings and approximately 100 text embeddings (depending on the training prompt). The same system prompt is used as the pre-trained vision model to maximize the effectiveness of transfer learning.
• Figure 4: ActionSense dataset. It consists of a variety of kitchen tasks, each with corresponding wearable sensor data, video data, and activity labels. Subjects perform a subset of these tasks over episodes of approximately 1 hour, with 10 total subjects.
• Figure 5: Comparison between Sensor2Text, GPT4, and fine-tuned VideoLLaMA. Each model demonstrates similar chat capabilities. However, Sensor2Text recalls the details of the subject's actions more precisely, demonstrating the advantages of using wearable sensors over video for some activities.