Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Vision Language Models for Dynamic Human Activity Recognition in Healthcare Settings

Abderrazek Abid, Thanh-Cong Ho, Fakhri Karray

TL;DR

The paper addresses HAR in remote healthcare by leveraging Vision-Language Models (VLMs) to produce descriptive activity captions, enabling natural-language queries while preserving privacy. It introduces a descriptive-caption dataset built from the Toyota Smarthome dataset and defines four evaluation methods to assess VLM HAR performance, including keyword matching and cosine-based semantic similarity. Experimental results show that VLMs can achieve performance comparable to or better than traditional deep-learning HAR models under several evaluation settings, highlighting potential for consolidating HAR with interactive clinical interfaces in intelligent healthcare systems. The work provides a strong benchmark and practical insights for deploying VLMs in RHMS, and suggests the dataset can facilitate fine-tuning and more rigorous evaluation of VLMs for open-ended HAR tasks.

Abstract

As generative AI continues to evolve, Vision Language Models (VLMs) have emerged as promising tools in various healthcare applications. One area that remains relatively underexplored is their use in human activity recognition (HAR) for remote health monitoring. VLMs offer notable strengths, including greater flexibility and the ability to overcome some of the constraints of traditional deep learning models. However, a key challenge in applying VLMs to HAR lies in the difficulty of evaluating their dynamic and often non-deterministic outputs. To address this gap, we introduce a descriptive caption data set and propose comprehensive evaluation methods to evaluate VLMs in HAR. Through comparative experiments with state-of-the-art deep learning models, our findings demonstrate that VLMs achieve comparable performance and, in some cases, even surpass conventional approaches in terms of accuracy. This work contributes a strong benchmark and opens new possibilities for the integration of VLMs into intelligent healthcare systems.

Vision Language Models for Dynamic Human Activity Recognition in Healthcare Settings

TL;DR

The paper addresses HAR in remote healthcare by leveraging Vision-Language Models (VLMs) to produce descriptive activity captions, enabling natural-language queries while preserving privacy. It introduces a descriptive-caption dataset built from the Toyota Smarthome dataset and defines four evaluation methods to assess VLM HAR performance, including keyword matching and cosine-based semantic similarity. Experimental results show that VLMs can achieve performance comparable to or better than traditional deep-learning HAR models under several evaluation settings, highlighting potential for consolidating HAR with interactive clinical interfaces in intelligent healthcare systems. The work provides a strong benchmark and practical insights for deploying VLMs in RHMS, and suggests the dataset can facilitate fine-tuning and more rigorous evaluation of VLMs for open-ended HAR tasks.

Abstract

As generative AI continues to evolve, Vision Language Models (VLMs) have emerged as promising tools in various healthcare applications. One area that remains relatively underexplored is their use in human activity recognition (HAR) for remote health monitoring. VLMs offer notable strengths, including greater flexibility and the ability to overcome some of the constraints of traditional deep learning models. However, a key challenge in applying VLMs to HAR lies in the difficulty of evaluating their dynamic and often non-deterministic outputs. To address this gap, we introduce a descriptive caption data set and propose comprehensive evaluation methods to evaluate VLMs in HAR. Through comparative experiments with state-of-the-art deep learning models, our findings demonstrate that VLMs achieve comparable performance and, in some cases, even surpass conventional approaches in terms of accuracy. This work contributes a strong benchmark and opens new possibilities for the integration of VLMs into intelligent healthcare systems.
Paper Structure (18 sections, 1 equation, 2 figures, 2 tables)

This paper contains 18 sections, 1 equation, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Remote Health Monitoring Systems
  4. Human Activity Recognition
  5. Human Activity Descriptive Caption Dataset
  6. Visual Datasets
  7. Methodology for Descriptive Caption Generation
  8. Experiments and Analysis
  9. Evaluation Methods
  10. Keyword Matching
  11. VLM-as-Judge
  12. BERTScore
  13. Cosine Similarity
  14. Experiment Settings
  15. Results and Discussion
  16. ...and 3 more sections

Figures (2)

  • Figure 1: Overview of the descriptive caption generation framework.
  • Figure 2: Comparison of InternVL-2.5 outputs and ground-truth captions