An LLM-Empowered Low-Resolution Vision System for On-Device Human Behavior Understanding

TL;DR

This work tackles on-device human behavior understanding from low-resolution sensors by addressing the modality and task gaps of existing LVLMs. It introduces Llambda, a three-stage system comprising a Contrastive-Oriented Data Labeler, a Physical-Knowledge-Guided Captioner, and LoRA-based efficient fine-tuning, to produce high-quality captions from depth, thermal, and infrared data with limited labels. Key innovations include window-based frame filtering, spatiotemporal consistency checks, contrastive learning for pseudo labels, and prompt-guided caption generation, all designed for edge deployment. Empirical evaluations on a region-scale real-world testbed and three low-resolution datasets show substantial improvements in Bert-Score (up to 87.02%) and a relative gain of about 40.03% over strong LVLM baselines, demonstrating practical potential for privacy-preserving, on-device HBU in healthcare and smart environments.

Abstract

The rapid advancements in Large Vision Language Models (LVLMs) offer the potential to surpass conventional labeling by generating richer, more detailed descriptions of on-device human behavior understanding (HBU) in low-resolution vision systems, such as depth, thermal, and infrared. However, existing large vision language model (LVLM) approaches are unable to understand low-resolution data well as they are primarily designed for high-resolution data, such as RGB images. A quick fixing approach is to caption a large amount of low-resolution data, but it requires a significant amount of labor-intensive annotation efforts. In this paper, we propose a novel, labor-saving system, Llambda, designed to support low-resolution HBU. The core idea is to leverage limited labeled data and a large amount of unlabeled data to guide LLMs in generating informative captions, which can be combined with raw data to effectively fine-tune LVLM models for understanding low-resolution videos in HBU. First, we propose a Contrastive-Oriented Data Labeler, which can capture behavior-relevant information from long, low-resolution videos and generate high-quality pseudo labels for unlabeled data via contrastive learning. Second, we propose a Physical-Knowledge Guided Captioner, which utilizes spatial and temporal consistency checks to mitigate errors in pseudo labels. Therefore, it can improve LLMs' understanding of sequential data and then generate high-quality video captions. Finally, to ensure on-device deployability, we employ LoRA-based efficient fine-tuning to adapt LVLMs for low-resolution data. We evaluate Llambda using a region-scale real-world testbed and three distinct low-resolution datasets, and the experiments show that Llambda outperforms several state-of-the-art LVLM systems up to $40.03\%$ on average Bert-Score.

Figures (16)

• Figure 1: Limitation of labels. The left depth frame illustrates a single-label classification, while the middle depth frame represents a multi-label classification. In contrast, the right depth frame provides a caption with a detailed description of the scene.
• Figure 2: Accuracy performance of the existing LVLMs. Existing single-modality LVLMs, such as Qwen2.5-VL-7B (Qwen) and InternVL2-8B (InternVL), as well as multi-modality LVLMs, such as OneLLM-7B (OneLLM) and LanguageBind-7B (LBind), demonstrate limited performance in understanding human behavior.
• Figure 3: Overview of Llambda. Llambda has three stages including Contrastive-Oriented Data Labeler, Physical-Knowledge-Guided Captioner, and LoRA-based Efficient Fine-tuning.
• Figure 4: Comparison of filtering approaches including Reducto li2020reducto, InFi yuan2022infi, YOLO yolov8_ultralytics in terms of error rate and computational workload (GFLOPs).
• Figure 5: Motivation of dynamic action capturing. We highlight the importance of focusing attention in action recognition using RGB and depth data. By applying a cropping mechanism, the model can better concentrate on human-relevant regions, reducing background noise and improving recognition performance.