Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An AI-Based System Utilizing IoT-Enabled Ambient Sensors and LLMs for Complex Activity Tracking

Yuan Sun, Jorge Ortiz

TL;DR

A non-invasive ambient sensing system that can detect multiple activities and apply large language models (LLMs) to reason the activity sequences to help elderly people in their daily activities, such as reminding them to take pills or handling emergencies like falls.

Abstract

Complex activity recognition plays an important role in elderly care assistance. However, the reasoning ability of edge devices is constrained by the classic machine learning model capacity. In this paper, we present a non-invasive ambient sensing system that can detect multiple activities and apply large language models (LLMs) to reason the activity sequences. This method effectively combines edge devices and LLMs to help elderly people in their daily activities, such as reminding them to take pills or handling emergencies like falls. The LLM-based edge device can also serve as an interface to interact with elderly people, especially with memory issue, assisting them in their daily lives. By deploying such a system, we believe that the smart sensing system can improve the quality of life for older people and provide more efficient protection

An AI-Based System Utilizing IoT-Enabled Ambient Sensors and LLMs for Complex Activity Tracking

TL;DR

A non-invasive ambient sensing system that can detect multiple activities and apply large language models (LLMs) to reason the activity sequences to help elderly people in their daily activities, such as reminding them to take pills or handling emergencies like falls.

Abstract

Complex activity recognition plays an important role in elderly care assistance. However, the reasoning ability of edge devices is constrained by the classic machine learning model capacity. In this paper, we present a non-invasive ambient sensing system that can detect multiple activities and apply large language models (LLMs) to reason the activity sequences. This method effectively combines edge devices and LLMs to help elderly people in their daily activities, such as reminding them to take pills or handling emergencies like falls. The LLM-based edge device can also serve as an interface to interact with elderly people, especially with memory issue, assisting them in their daily lives. By deploying such a system, we believe that the smart sensing system can improve the quality of life for older people and provide more efficient protection
Paper Structure (8 sections, 7 figures, 1 table)

This paper contains 8 sections, 7 figures, 1 table.

Table of Contents

  1. Introduction
  2. LLM and Complex Reasoning
  3. sensor board setup
  4. initial experiment
  5. Data Collection
  6. Initial Results
  7. Applying LLMs to Sequence Detection and Complex Activity Recognition
  8. Conclusion and Future Work

Figures (7)

  • Figure 1: The system employs both local inference using ambient sensors and reasoning via a cloud-based LLM. The sensors detect atomic activities, and the cloud server receives these activity sequences as context to further detect higher-level meanings, make decisions, and interact with the user.
  • Figure 2: The non-intrusive sensor board we design for our system
  • Figure 3: Raspberry Pi Model B+ used in our ambient sensor setup, facilitating seamless integration for elderly care assistance and activity tracking
  • Figure 4: Eating activity collected by the ambient sensor
  • Figure 5: Data collected from chopping activities
  • ...and 2 more figures