PainFormer: a Vision Foundation Model for Automatic Pain Assessment

TL;DR

PainFormer tackles automatic pain assessment across multiple behavioral and physiological modalities by presenting a vision foundation model trained in a multi-task fashion on 14 datasets (~10.9M samples), with embedding extraction followed by a dedicated Embedding-Mixer and Video-Encoder to enable versatile unimodal and multimodal pipelines. The approach delivers state-of-the-art performance on BioVid and AI4Pain across RGB, synthetic thermal, depth videos, and biosignals (ECG, EMG, GSR, fNIRS), and provides attention-based interpretability via attention maps. Key contributions include the use of synthetic thermal and depth modalities, extensive cross-modal evaluation, and strong fusion results that demonstrate robustness and practicality for clinical pain assessment. The work underscores the value of foundation-models and multi-modality data fusion, showing clear potential for scalable, privacy-conscious, real-world pain monitoring and management.

Abstract

Pain is a manifold condition that impacts a significant percentage of the population. Accurate and reliable pain evaluation for the people suffering is crucial to developing effective and advanced pain management protocols. Automatic pain assessment systems provide continuous monitoring and support decision-making processes, ultimately aiming to alleviate distress and prevent functionality decline. This study introduces PainFormer, a vision foundation model based on multi-task learning principles trained simultaneously on 14 tasks/datasets with a total of 10.9 million samples. Functioning as an embedding extractor for various input modalities, the foundation model provides feature representations to the Embedding-Mixer, a transformer-based module that performs the final pain assessment. Extensive experiments employing behavioral modalities - including RGB, synthetic thermal, and estimated depth videos - and physiological modalities such as ECG, EMG, GSR, and fNIRS revealed that PainFormer effectively extracts high-quality embeddings from diverse input modalities. The proposed framework is evaluated on two pain datasets, BioVid and AI4Pain, and directly compared to 75 different methodologies documented in the literature. Experiments conducted in unimodal and multimodal settings demonstrate state-of-the-art performances across modalities and pave the way toward general-purpose models for automatic pain assessment. The foundation model's architecture (code) and weights are available at: https://github.com/GkikasStefanos/PainFormer.

Figures (5)

• Figure 1: Representation of the main models and their components presented in this study: (a)PainFormer is organized hierarchically into four stages, integrating Spectral and Self-Attention Layers to function as the embedding extractor for the inputs; (b) The Spectral Layer, a primary component of PainFormer, applies FFT to compute frequency-related information combined with a learnable filter $K$ to emphasize important frequencies; (c) The Self-Attention Layer, another primary module of PainFormer, facilitates parallel computation of features and their relationship; (d) The Embedding-Mixer, which combines cross and self-attention mechanisms, serves as the module for final classification of the embeddings used in the pain assessment task; (e) The Video-Encoder, a compact and efficient module, encodes video representations into a lower dimensional space; (f) The MLP-1 is included in the Spectral Layer; (g) The MLP-2, part of the Self-Attention Layer; (h) The MLP-3 structure is incorporated within the Embedding-Mixer and Video-Encoder.
• Figure 2: Examples of different vision modalities in frame samples: (a) RGB frame, (b) synthetic thermal frame, and (c) depth estimation frame.
• Figure 3: Examples of different visual representations for biosignals: (a)waveform, (b)spectrogram-angle, (c)spectrogram-phase, and (d)spectrogram-PSD.
• Figure 4: A high-level overview of the presented framework. PainFormer, the foundation model, is capable of extracting high-quality embeddings from a wide range of different behavioral and physiological modalities. Evaluating RGB, thermal, and depth videos and various representations of ECG, EMG, GSR, and fNIRS, including waveforms and spectrograms, demonstrate the comprehensive information encapsulated within these embeddings. Utilizing the embeddings from the PainFormer enables the development of diverse unimodal and multimodal pipelines for the pain assessment task. Each pipeline can be customized according to the modalities used, dataset characteristics, and the requirements of the target application or clinical environment. Our evaluations involved developing and applying various pipelines in unimodal and multimodal settings, achieving state-of-the-art results across different modalities and data representations.
• Figure 5: Attention maps from the PainFormer: (a)(1st row) frames from RGB, thermal, and depth video modalities; (a)(2nd row) corresponding attention maps; (b)(1st row) attention maps for ECG and EMG; (b)(2nd row) attention maps for EDA and fNIRS modalities.