Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ContextGPT: Infusing LLMs Knowledge into Neuro-Symbolic Activity Recognition Models

Luca Arrotta, Claudio Bettini, Gabriele Civitarese, Michele Fiori

TL;DR

ContextGPT tackles the data scarcity challenge in context-aware HAR by using prompt-engineered LLMs to extract common-sense knowledge about activity-context relationships and infusing it into a Neuro-Symbolic HAR model. By replacing hand-crafted ontologies with LLM-derived knowledge and employing a NIMBUS-inspired infusion, the approach reduces expert engineering while maintaining competitive performance. Experiments on DOMINO and ExtraSensory demonstrate that ContextGPT achieves results comparable to ontology-based baselines under data-scarce conditions, with notable gains in efficiency. The work suggests a practical, scalable path toward broader adoption of context-aware HAR with LLM-assisted knowledge infusion, and points to future directions in domain specialization and edge deployments.

Abstract

Context-aware Human Activity Recognition (HAR) is a hot research area in mobile computing, and the most effective solutions in the literature are based on supervised deep learning models. However, the actual deployment of these systems is limited by the scarcity of labeled data that is required for training. Neuro-Symbolic AI (NeSy) provides an interesting research direction to mitigate this issue, by infusing common-sense knowledge about human activities and the contexts in which they can be performed into HAR deep learning classifiers. Existing NeSy methods for context-aware HAR rely on knowledge encoded in logic-based models (e.g., ontologies) whose design, implementation, and maintenance to capture new activities and contexts require significant human engineering efforts, technical knowledge, and domain expertise. Recent works show that pre-trained Large Language Models (LLMs) effectively encode common-sense knowledge about human activities. In this work, we propose ContextGPT: a novel prompt engineering approach to retrieve from LLMs common-sense knowledge about the relationship between human activities and the context in which they are performed. Unlike ontologies, ContextGPT requires limited human effort and expertise. An extensive evaluation carried out on two public datasets shows how a NeSy model obtained by infusing common-sense knowledge from ContextGPT is effective in data scarcity scenarios, leading to similar (and sometimes better) recognition rates than logic-based approaches with a fraction of the effort.

ContextGPT: Infusing LLMs Knowledge into Neuro-Symbolic Activity Recognition Models

TL;DR

ContextGPT tackles the data scarcity challenge in context-aware HAR by using prompt-engineered LLMs to extract common-sense knowledge about activity-context relationships and infusing it into a Neuro-Symbolic HAR model. By replacing hand-crafted ontologies with LLM-derived knowledge and employing a NIMBUS-inspired infusion, the approach reduces expert engineering while maintaining competitive performance. Experiments on DOMINO and ExtraSensory demonstrate that ContextGPT achieves results comparable to ontology-based baselines under data-scarce conditions, with notable gains in efficiency. The work suggests a practical, scalable path toward broader adoption of context-aware HAR with LLM-assisted knowledge infusion, and points to future directions in domain specialization and edge deployments.

Abstract

Context-aware Human Activity Recognition (HAR) is a hot research area in mobile computing, and the most effective solutions in the literature are based on supervised deep learning models. However, the actual deployment of these systems is limited by the scarcity of labeled data that is required for training. Neuro-Symbolic AI (NeSy) provides an interesting research direction to mitigate this issue, by infusing common-sense knowledge about human activities and the contexts in which they can be performed into HAR deep learning classifiers. Existing NeSy methods for context-aware HAR rely on knowledge encoded in logic-based models (e.g., ontologies) whose design, implementation, and maintenance to capture new activities and contexts require significant human engineering efforts, technical knowledge, and domain expertise. Recent works show that pre-trained Large Language Models (LLMs) effectively encode common-sense knowledge about human activities. In this work, we propose ContextGPT: a novel prompt engineering approach to retrieve from LLMs common-sense knowledge about the relationship between human activities and the context in which they are performed. Unlike ontologies, ContextGPT requires limited human effort and expertise. An extensive evaluation carried out on two public datasets shows how a NeSy model obtained by infusing common-sense knowledge from ContextGPT is effective in data scarcity scenarios, leading to similar (and sometimes better) recognition rates than logic-based approaches with a fraction of the effort.
Paper Structure (26 sections, 6 figures)

This paper contains 26 sections, 6 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Data scarcity in HAR
  4. Neuro-Symbolic HAR
  5. Using LLMs for Human Activity Recognition
  6. Methodology
  7. A Neuro-Symbolic Framework for Context-Aware HAR
  8. ContextGPT architecture
  9. Prompt Engineering
  10. System Message
  11. Context2Text
  12. Example pool
  13. Example Selection
  14. Post-processing
  15. Infusing knowledge into Neuro-Symbolic model
  16. ...and 11 more sections

Figures (6)

  • Figure 1: A Neuro-Symbolic AI framework for context-aware HAR gathering knowledge from ContextGPT
  • Figure 2: Overall architecture of ContextGPT
  • Figure 3: The system message of ContextGPT. The possible activities, in this case, are the ones of the DOMINO arrotta2023domino dataset.
  • Figure 4: An example of LLM output
  • Figure 5: Infusing ContextGPT into the Symbolic Features approach
  • ...and 1 more figures