Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reducing Annotation Burden in Physical Activity Research Using Vision-Language Models

Abram Schonfeldt, Benjamin Maylor, Xiaofang Chen, Ronald Clark, Aiden Doherty

TL;DR

Reducing annotation burden in physical activity research using vision-language models investigates whether open-source vision-language models (VLMs) and discriminative models can predict activity-intensity classes from wearable camera images in two large free-living cohorts. The study compares dual-encoder CLIP and generative BLIP2/LLaVA against standard discriminative baselines, finding strong sedentary-behaviour predictions in the Oxfordshire cohort but substantial generalization gaps to the Sichuan cohort, with SB being the most reliably detected class. Key findings show that freely available VLMs can help annotate sedentary behaviour and reduce annotation workload in similar populations, though cross-population generalization and low-frame-rate issues remain significant challenges. The authors advocate using the best-performing models to label SB in free-living studies and suggest upgrading to video-capable wearables to improve ground-truth reliability, while outlining avenues for continual learning and uncertainty quantification to enhance robustness.

Abstract

Introduction: Data from wearable devices collected in free-living settings, and labelled with physical activity behaviours compatible with health research, are essential for both validating existing wearable-based measurement approaches and developing novel machine learning approaches. One common way of obtaining these labels relies on laborious annotation of sequences of images captured by cameras worn by participants through the course of a day. Methods: We compare the performance of three vision language models and two discriminative models on two free-living validation studies with 161 and 111 participants, collected in Oxfordshire, United Kingdom and Sichuan, China, respectively, using the Autographer (OMG Life, defunct) wearable camera. Results: We found that the best open-source vision-language model (VLM) and fine-tuned discriminative model (DM) achieved comparable performance when predicting sedentary behaviour from single images on unseen participants in the Oxfordshire study; median F1-scores: VLM = 0.89 (0.84, 0.92), DM = 0.91 (0.86, 0.95). Performance declined for light (VLM = 0.60 (0.56,0.67), DM = 0.70 (0.63, 0.79)), and moderate-to-vigorous intensity physical activity (VLM = 0.66 (0.53, 0.85); DM = 0.72 (0.58, 0.84)). When applied to the external Sichuan study, performance fell across all intensity categories, with median Cohen's kappa-scores falling from 0.54 (0.49, 0.64) to 0.26 (0.15, 0.37) for the VLM, and from 0.67 (0.60, 0.74) to 0.19 (0.10, 0.30) for the DM. Conclusion: Freely available computer vision models could help annotate sedentary behaviour, typically the most prevalent activity of daily living, from wearable camera images within similar populations to seen data, reducing the annotation burden.

Reducing Annotation Burden in Physical Activity Research Using Vision-Language Models

TL;DR

Reducing annotation burden in physical activity research using vision-language models investigates whether open-source vision-language models (VLMs) and discriminative models can predict activity-intensity classes from wearable camera images in two large free-living cohorts. The study compares dual-encoder CLIP and generative BLIP2/LLaVA against standard discriminative baselines, finding strong sedentary-behaviour predictions in the Oxfordshire cohort but substantial generalization gaps to the Sichuan cohort, with SB being the most reliably detected class. Key findings show that freely available VLMs can help annotate sedentary behaviour and reduce annotation workload in similar populations, though cross-population generalization and low-frame-rate issues remain significant challenges. The authors advocate using the best-performing models to label SB in free-living studies and suggest upgrading to video-capable wearables to improve ground-truth reliability, while outlining avenues for continual learning and uncertainty quantification to enhance robustness.

Abstract

Introduction: Data from wearable devices collected in free-living settings, and labelled with physical activity behaviours compatible with health research, are essential for both validating existing wearable-based measurement approaches and developing novel machine learning approaches. One common way of obtaining these labels relies on laborious annotation of sequences of images captured by cameras worn by participants through the course of a day. Methods: We compare the performance of three vision language models and two discriminative models on two free-living validation studies with 161 and 111 participants, collected in Oxfordshire, United Kingdom and Sichuan, China, respectively, using the Autographer (OMG Life, defunct) wearable camera. Results: We found that the best open-source vision-language model (VLM) and fine-tuned discriminative model (DM) achieved comparable performance when predicting sedentary behaviour from single images on unseen participants in the Oxfordshire study; median F1-scores: VLM = 0.89 (0.84, 0.92), DM = 0.91 (0.86, 0.95). Performance declined for light (VLM = 0.60 (0.56,0.67), DM = 0.70 (0.63, 0.79)), and moderate-to-vigorous intensity physical activity (VLM = 0.66 (0.53, 0.85); DM = 0.72 (0.58, 0.84)). When applied to the external Sichuan study, performance fell across all intensity categories, with median Cohen's kappa-scores falling from 0.54 (0.49, 0.64) to 0.26 (0.15, 0.37) for the VLM, and from 0.67 (0.60, 0.74) to 0.19 (0.10, 0.30) for the DM. Conclusion: Freely available computer vision models could help annotate sedentary behaviour, typically the most prevalent activity of daily living, from wearable camera images within similar populations to seen data, reducing the annotation burden.
Paper Structure (25 sections, 1 equation, 5 figures, 7 tables)

This paper contains 25 sections, 1 equation, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Relevant work
  3. Wearable data-sets of health-relevant behaviours
  4. CAPTURE24: the Oxfordshire and Sichuan studies
  5. Recognising activities from sparse sequences of egocentric images
  6. Vision language models
  7. Methods
  8. Data processing and quality assessment
  9. Simplifying labels
  10. Predicting activity intensity using computer vision
  11. Discriminative models
  12. Dual-encoder CLIP
  13. Generative models
  14. Evaluation
  15. Results
  16. ...and 10 more sections

Figures (5)

  • Figure 1: Illustration of the computer vision approaches compared (top). Below, quartile plots tufte_visual_2002 show the five-number summary of per-participant F$_1$-scores for Sedentary Behaviour (SB), Light Intensity Physical Activity (LIPA), and Moderate-to-Vigorous Physical Activity (MVPA), for the best-performing vision-language model, LLaVA (squares), and the best-performing discriminative vision model, ViT (circles), selected via hyperparameter tuning. Performance is shown for participants in the Oxfordshire study (blue) and the Sichuan study (red) withheld from model selection. MVPA constitutes only 8% of the training set, which is reflected in the high variance of per-participant F$_1$-scores.
  • Figure 2: Impact of different hyperparameters on the performance of each model on the validation-set of the Oxfordshire study.
  • Figure 3: A sequence of images captured with an interval of 20 seconds between frames, labelled with activities and MET values.
  • Figure 4: Visualisation of the temporal sparsity of images, the label imbalance, and the large number of obscure images in the Oxfordshire and Sichuan validation study. The median participants day has 100 labelled events in the Oxfordshire study, versus 50 in the Sichuan study, with the much lower capture rate in this study potentially limiting the number of events that could be labelled. The majority of images were labelled as depicting sedentary activity.
  • Figure 5: Confusion matrices showing the disagreements between the human and model predictions, for LLaVA and ViT, particularly on the Sichuan study. The percentages (and colours) are normalised based on the total number of "true" instances of each label.