PhysLLM: Harnessing Large Language Models for Cross-Modal Remote Physiological Sensing

TL;DR

PhysLLM addresses the fragility of non-contact rPPG measurements under illumination changes and motion by coupling a CNN-based rPPG backbone with a large language model through a Text Prototype Guidance framework and a Dual-Domain Stationary algorithm. It introduces physiological cue-aware prompting and adaptive cross-modal fusion (via a Vision Aggregator and TPG) to align hemodynamic signals with linguistic tokens, enabling long-range temporal reasoning and robust performance across four benchmark datasets. The approach demonstrates strong intra- and cross-dataset generalization and provides interpretable cross-modal attention visualizations, highlighting attention to physiologically relevant skin regions. These results advance non-contact physiological sensing by enabling robust, context-aware estimation in challenging real-world conditions and point toward scalable, edge-friendly multi-modal health monitoring solutions.

Abstract

Remote photoplethysmography (rPPG) enables non-contact physiological measurement but remains highly susceptible to illumination changes, motion artifacts, and limited temporal modeling. Large Language Models (LLMs) excel at capturing long-range dependencies, offering a potential solution but struggle with the continuous, noise-sensitive nature of rPPG signals due to their text-centric design. To bridge this gap, we introduce PhysLLM, a collaborative optimization framework that synergizes LLMs with domain-specific rPPG components. Specifically, the Text Prototype Guidance (TPG) strategy is proposed to establish cross-modal alignment by projecting hemodynamic features into LLM-interpretable semantic space, effectively bridging the representational gap between physiological signals and linguistic tokens. Besides, a novel Dual-Domain Stationary (DDS) Algorithm is proposed for resolving signal instability through adaptive time-frequency domain feature re-weighting. Finally, rPPG task-specific cues systematically inject physiological priors through physiological statistics, environmental contextual answering, and task description, leveraging cross-modal learning to integrate both visual and textual information, enabling dynamic adaptation to challenging scenarios like variable illumination and subject movements. Evaluation on four benchmark datasets, PhysLLM achieves state-of-the-art accuracy and robustness, demonstrating superior generalization across lighting variations and motion scenarios.

Figures (5)

• Figure 1: The comparison between (a) typical CNN-based rPPG model, (b) pure LLM model, and (c) our proposed PhysLLM. And (d) is the performance comparison of different methods on the UBFC-rPPG bobbia2019unsupervised dataset.
• Figure 2: Framework of the PhysLLM. The architecture of PhysLLM comprises three principal data streams that operate in concert. The leftmost stream represents the Physiological Cue-Aware Prompt Learning module, which incorporates task-specific prior knowledge through adaptive prompt learning while generating context-aware prompt tokens. The central and rightmost streams collectively form the Text-Vision-Sequence Embedding Generation pipeline, which integrates our novel Dual-Domain Stationary (DDS) Algorithm and Text Prototype Guidance (TPG) module. This integrated approach facilitates the extraction of sequence tokens from temporal physiological data and visual tokens from facial imagery, both guided by LLM-generated text prototypes that serve as semantic anchors for cross-modal alignment.
• Figure 3: Architecture of the Vision Aggregator. It employs a hierarchical attention architecture to dynamically synthesize multi-scale feature representations.
• Figure 4: Introduction to the composition of cues. (a) Extracting visual priors (e.g., lighting, facial expressions, occlusions) via LLaVA and encoding them into visual tokens. (b) Tokenizing textual descriptions of the rPPG task to derive task-specific priors. (c) Analyzing statistical features of rPPG signals from the backbone network to generate statistical prior tokens. These cues synthesize visual, semantic, and statistical priors for enhanced physiological signal analysis.
• Figure 5: Visualization of saliency maps from PhysLLM on PURE stricker2014non and UBFC-rPPG bobbia2019unsupervised.