Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AdCare-VLM: Towards a Unified and Pre-aligned Latent Representation for Healthcare Video Understanding

Md Asaduzzaman Jabin, Hanqi Jiang, Yiwei Li, Patrick Kaggwa, Eugene Douglass, Juliet N. Sekandi, Tianming Liu

TL;DR

AdCare-VLM addresses the critical problem of medication adherence monitoring by unifying visual and linguistic representations in a LVLM framework. It introduces a pre-aligned, shared latent space for images and videos, enabling robust video-based VQA on adherence through a Vicuna-backed model with LanguageBind and a learnable projection. Trained on a private TB dataset (806 videos) and evaluated on the LLM-TB-VQA benchmark, the approach demonstrates consistent improvements over PEFT-based baselines across pre-trained, regular, and LoRA configurations, with ablations and attention maps supporting interpretability. The work highlights the potential of unified visual-language representations to automate tele-medication monitoring in resource-limited settings, with implications for reducing clinician workload and improving adherence outcomes, while also releasing a pathway for future bias-aware, data-efficient expansion to additional modalities and diseases.

Abstract

Chronic diseases, including diabetes, hypertension, asthma, HIV-AIDS, epilepsy, and tuberculosis, necessitate rigorous adherence to medication to avert disease progression, manage symptoms, and decrease mortality rates. Adherence is frequently undermined by factors including patient behavior, caregiver support, elevated medical costs, and insufficient healthcare infrastructure. We propose AdCare-VLM, a specialized LLaVA-based multimodal large vision language model (LVLM) by introducing a unified visual latent space with pre-alignment to facilitate visual question answering (VQA) concerning medication adherence through patient videos. We employ a private dataset comprising 806 custom-annotated tuberculosis (TB) medication monitoring videos, which have been labeled by clinical experts, to fine-tune the model for adherence pattern detection. We present LLM-TB-VQA, a detailed medical adherence VQA dataset that encompasses positive, negative, and ambiguous adherence cases. Our method identifies correlations between visual features, such as the clear visibility of the patient's face, medication, water intake, and the act of ingestion, and their associated medical concepts in captions. This facilitates the integration of aligned visual-linguistic representations and improves multimodal interactions. Experimental results indicate that our method surpasses parameter-efficient fine-tuning (PEFT) enabled VLM models, such as LLaVA-V1.5 and Chat-UniVi, with absolute improvements ranging from 3.1% to 3.54% across pre-trained, regular, and low-rank adaptation (LoRA) configurations. Comprehensive ablation studies and attention map visualizations substantiate our approach, enhancing interpretability.

AdCare-VLM: Towards a Unified and Pre-aligned Latent Representation for Healthcare Video Understanding

TL;DR

AdCare-VLM addresses the critical problem of medication adherence monitoring by unifying visual and linguistic representations in a LVLM framework. It introduces a pre-aligned, shared latent space for images and videos, enabling robust video-based VQA on adherence through a Vicuna-backed model with LanguageBind and a learnable projection. Trained on a private TB dataset (806 videos) and evaluated on the LLM-TB-VQA benchmark, the approach demonstrates consistent improvements over PEFT-based baselines across pre-trained, regular, and LoRA configurations, with ablations and attention maps supporting interpretability. The work highlights the potential of unified visual-language representations to automate tele-medication monitoring in resource-limited settings, with implications for reducing clinician workload and improving adherence outcomes, while also releasing a pathway for future bias-aware, data-efficient expansion to additional modalities and diseases.

Abstract

Chronic diseases, including diabetes, hypertension, asthma, HIV-AIDS, epilepsy, and tuberculosis, necessitate rigorous adherence to medication to avert disease progression, manage symptoms, and decrease mortality rates. Adherence is frequently undermined by factors including patient behavior, caregiver support, elevated medical costs, and insufficient healthcare infrastructure. We propose AdCare-VLM, a specialized LLaVA-based multimodal large vision language model (LVLM) by introducing a unified visual latent space with pre-alignment to facilitate visual question answering (VQA) concerning medication adherence through patient videos. We employ a private dataset comprising 806 custom-annotated tuberculosis (TB) medication monitoring videos, which have been labeled by clinical experts, to fine-tune the model for adherence pattern detection. We present LLM-TB-VQA, a detailed medical adherence VQA dataset that encompasses positive, negative, and ambiguous adherence cases. Our method identifies correlations between visual features, such as the clear visibility of the patient's face, medication, water intake, and the act of ingestion, and their associated medical concepts in captions. This facilitates the integration of aligned visual-linguistic representations and improves multimodal interactions. Experimental results indicate that our method surpasses parameter-efficient fine-tuning (PEFT) enabled VLM models, such as LLaVA-V1.5 and Chat-UniVi, with absolute improvements ranging from 3.1% to 3.54% across pre-trained, regular, and low-rank adaptation (LoRA) configurations. Comprehensive ablation studies and attention map visualizations substantiate our approach, enhancing interpretability.
Paper Structure (17 sections, 9 equations, 5 figures, 7 tables)

This paper contains 17 sections, 9 equations, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Data Source, Population and Annotation
  3. Related Work
  4. Medical vision question answering (VQA)
  5. Large Vision Language Approach
  6. The Model Architecture
  7. Framework Overview
  8. United Visual Latent Space
  9. Pre-Alignment to Projection
  10. Pipeline
  11. Understand Pre-Training
  12. Fine-tuning
  13. Experiments and Results
  14. System Details
  15. Evaluation Process
  16. ...and 2 more sections

Figures (5)

  • Figure 1: Architecture: AdCare-VLM utilizes LanguageBind and a multimodal language model for video processing. The visual encoder extracts spatio-temporal features by averaging frame-level characteristics across both temporal and spatial dimensions. Following pre alignment, these features are mapped into the input space of the LLM through a learnable Multi-Layer Perceptron (MLP), establishing a multimodal pipeline designed for effective video-based question answering in the context of medication adherence monitoring.
  • Figure A1: The architectural overview of image and video pre-alignment prior to projection is provided to aid in the creation of a unified representation.
  • Figure A2: AdCare-VLM-7B Model with LoRA fine-tuning on qualitative outcomes (A selection of key frames is presented (blurred to protect subject’s privacy), emphasizing critical features for positive medication adherence, as indicated by the model, with red-underlined text).
  • Figure A3: AdCare-VLM-7B Model with LoRA fine-tuning on qualitative outcomes (A selection of key frames is presented (blurred to protect subject’s privacy), emphasizing critical features for negative medication adherence, as indicated by the model, with red-underlined text).
  • Figure A4: AdCare-VLM-7B Model with LoRA fine-tuning on qualitative outcomes (A selection of key frames is presented (blurred to protect subject’s privacy), emphasizing critical features for ambiguous medication adherence, as indicated by the model, with red-underlined text).