Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On-device Large Multi-modal Agent for Human Activity Recognition

Md Shakhrul Iman Siam, Ishtiaque Ahmed Showmik, Guanqun Song, Ting Zhu

TL;DR

This work addresses the challenge of robust, on-device human activity recognition (HAR) by introducing a Large Multi-Modal Agent that leverages large language models (LLMs) to classify activities from IMU time-series data while providing reasoning and interactive Q&A. The design converts raw sensor streams into compact statistical features and aligns them with text-based inputs to enable on-device LLM inference using LoRA-based fine-tuning and instruction tuning. Empirical results across Shoaib, HHAR, UCI HAR, MotionSense, and WISDM show competitive accuracy relative to state-of-the-art baselines and improved interpretability, with the LLM approach excelling more in unseen data scenarios. The paper also discusses modality alignment challenges and proposes encoder-based feature extraction as a future enhancement to further improve accuracy and efficiency on edge devices.

Abstract

Human Activity Recognition (HAR) has been an active area of research, with applications ranging from healthcare to smart environments. The recent advancements in Large Language Models (LLMs) have opened new possibilities to leverage their capabilities in HAR, enabling not just activity classification but also interpretability and human-like interaction. In this paper, we present a Large Multi-Modal Agent designed for HAR, which integrates the power of LLMs to enhance both performance and user engagement. The proposed framework not only delivers activity classification but also bridges the gap between technical outputs and user-friendly insights through its reasoning and question-answering capabilities. We conduct extensive evaluations using widely adopted HAR datasets, including HHAR, Shoaib, Motionsense to assess the performance of our framework. The results demonstrate that our model achieves high classification accuracy comparable to state-of-the-art methods while significantly improving interpretability through its reasoning and Q&A capabilities.

On-device Large Multi-modal Agent for Human Activity Recognition

TL;DR

This work addresses the challenge of robust, on-device human activity recognition (HAR) by introducing a Large Multi-Modal Agent that leverages large language models (LLMs) to classify activities from IMU time-series data while providing reasoning and interactive Q&A. The design converts raw sensor streams into compact statistical features and aligns them with text-based inputs to enable on-device LLM inference using LoRA-based fine-tuning and instruction tuning. Empirical results across Shoaib, HHAR, UCI HAR, MotionSense, and WISDM show competitive accuracy relative to state-of-the-art baselines and improved interpretability, with the LLM approach excelling more in unseen data scenarios. The paper also discusses modality alignment challenges and proposes encoder-based feature extraction as a future enhancement to further improve accuracy and efficiency on edge devices.

Abstract

Human Activity Recognition (HAR) has been an active area of research, with applications ranging from healthcare to smart environments. The recent advancements in Large Language Models (LLMs) have opened new possibilities to leverage their capabilities in HAR, enabling not just activity classification but also interpretability and human-like interaction. In this paper, we present a Large Multi-Modal Agent designed for HAR, which integrates the power of LLMs to enhance both performance and user engagement. The proposed framework not only delivers activity classification but also bridges the gap between technical outputs and user-friendly insights through its reasoning and question-answering capabilities. We conduct extensive evaluations using widely adopted HAR datasets, including HHAR, Shoaib, Motionsense to assess the performance of our framework. The results demonstrate that our model achieves high classification accuracy comparable to state-of-the-art methods while significantly improving interpretability through its reasoning and Q&A capabilities.
Paper Structure (26 sections, 4 equations, 15 figures, 3 tables)

This paper contains 26 sections, 4 equations, 15 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Sensing for Smart Health
  4. Efficient IoT Infrastructure for HAR
  5. Security and Trustworthiness in Sensing
  6. Large Language Models for HAR
  7. Design
  8. Dataset
  9. Shoaib Dataset
  10. Heterogeneity Dataset for Human Activity Recognition (HHAR)
  11. UCI Human Activity Recognition (HAR) Using Smartphones Dataset
  12. MotionSense Dataset
  13. WISDM Dataset
  14. Data Preprocessing
  15. Zero-shot capability of LLMs
  16. ...and 11 more sections

Figures (15)

  • Figure 1: The proposed framework is capable of classifying Human activity, providing reasoning, and performing QnA tasks.
  • Figure 2: Human activity labels in different datasets.
  • Figure 3: Correlation Analysis.
  • Figure 4: PDF Distributions of Sensor Features.
  • Figure 5: PCA analysis for Shoaib Dataset
  • ...and 10 more figures