Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Frugal Federated Learning for Violence Detection: A Comparison of LoRA-Tuned VLMs and Personalized CNNs

Sébastien Thuau, Siba Haidar, Ayush Bajracharya, Rachid Chelouah

TL;DR

This work investigates privacy-preserving violence detection under non-IID federated settings by comparing frugal LoRA-tuned vision-language models with a personalized CNN3D. It provides a systematic, energy- and emissions-aware evaluation across zero-shot, federated LoRA fine-tuning, and personalized federated learning, highlighting a surprising efficiency gap: a lightweight CNN3D can match or exceed VLM-based methods on key metrics while consuming substantially less energy and emitting fewer CO$_2$e. The findings argue for a hybrid deployment: default to efficient CNNs for routine classification and selectively activate VLMs for complex, context-rich inferences, enabling resource-aware, privacy-preserving surveillance. The study extends beyond accuracy to lifecycle considerations, offering a reproducible baseline for sustainable, multimodal, real-time video surveillance in line with regulatory and environmental imperatives.

Abstract

We examine frugal federated learning approaches to violence detection by comparing two complementary strategies: (i) zero-shot and federated fine-tuning of vision-language models (VLMs), and (ii) personalized training of a compact 3D convolutional neural network (CNN3D). Using LLaVA-7B and a 65.8M parameter CNN3D as representative cases, we evaluate accuracy, calibration, and energy usage under realistic non-IID settings. Both approaches exceed 90% accuracy. CNN3D slightly outperforms Low-Rank Adaptation(LoRA)-tuned VLMs in ROC AUC and log loss, while using less energy. VLMs remain favorable for contextual reasoning and multimodal inference. We quantify energy and CO$_2$ emissions across training and inference, and analyze sustainability trade-offs for deployment. To our knowledge, this is the first comparative study of LoRA-tuned vision-language models and personalized CNNs for federated violence detection, with an emphasis on energy efficiency and environmental metrics. These findings support a hybrid model: lightweight CNNs for routine classification, with selective VLM activation for complex or descriptive scenarios. The resulting framework offers a reproducible baseline for responsible, resource-aware AI in video surveillance, with extensions toward real-time, multimodal, and lifecycle-aware systems.

Frugal Federated Learning for Violence Detection: A Comparison of LoRA-Tuned VLMs and Personalized CNNs

TL;DR

This work investigates privacy-preserving violence detection under non-IID federated settings by comparing frugal LoRA-tuned vision-language models with a personalized CNN3D. It provides a systematic, energy- and emissions-aware evaluation across zero-shot, federated LoRA fine-tuning, and personalized federated learning, highlighting a surprising efficiency gap: a lightweight CNN3D can match or exceed VLM-based methods on key metrics while consuming substantially less energy and emitting fewer COe. The findings argue for a hybrid deployment: default to efficient CNNs for routine classification and selectively activate VLMs for complex, context-rich inferences, enabling resource-aware, privacy-preserving surveillance. The study extends beyond accuracy to lifecycle considerations, offering a reproducible baseline for sustainable, multimodal, real-time video surveillance in line with regulatory and environmental imperatives.

Abstract

We examine frugal federated learning approaches to violence detection by comparing two complementary strategies: (i) zero-shot and federated fine-tuning of vision-language models (VLMs), and (ii) personalized training of a compact 3D convolutional neural network (CNN3D). Using LLaVA-7B and a 65.8M parameter CNN3D as representative cases, we evaluate accuracy, calibration, and energy usage under realistic non-IID settings. Both approaches exceed 90% accuracy. CNN3D slightly outperforms Low-Rank Adaptation(LoRA)-tuned VLMs in ROC AUC and log loss, while using less energy. VLMs remain favorable for contextual reasoning and multimodal inference. We quantify energy and CO emissions across training and inference, and analyze sustainability trade-offs for deployment. To our knowledge, this is the first comparative study of LoRA-tuned vision-language models and personalized CNNs for federated violence detection, with an emphasis on energy efficiency and environmental metrics. These findings support a hybrid model: lightweight CNNs for routine classification, with selective VLM activation for complex or descriptive scenarios. The resulting framework offers a reproducible baseline for responsible, resource-aware AI in video surveillance, with extensions toward real-time, multimodal, and lifecycle-aware systems.
Paper Structure (25 sections, 1 figure, 6 tables)

This paper contains 25 sections, 1 figure, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Privacy-Preserving Architectures for Video Surveillance
  4. LoRA Fine-Tuning in Federated Learning
  5. Energy-Efficient Multimodal AI Models
  6. Environmental Impact Assessment
  7. Methodology
  8. Overview
  9. Datasets and Client Partitioning
  10. Federated Learning Setup
  11. LoRA-Based Fine-Tuning: Centralized vs. Federated
  12. Personalized CNN3D with Parameter Decoupling
  13. Evaluation Protocol: Performance and Environmental Metrics
  14. Experiments and Results
  15. Exp. 1 : Zero-Shot VLM Inference: Accuracy vs. Efficiency
  16. ...and 10 more sections

Figures (1)

  • Figure 1: Taxonomy of LoRA-based federated learning approaches. Methods are grouped into (left) aggregation-side coordination (LoRA-FAIR, FLORA) and (right) client-side adapter tuning (FDLoRA, FLoCoRA, HAFL). Our implementation follows the adapter-tuning path with standard LoRA modules on VLMs. The diagram is conceptual (no ranking implied).