Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Test-Time Adaptation via Cache Personalization for Facial Expression Recognition in Videos

Masoumeh Sharafi, Muhammad Osama Zeeshan, Soufiane Belharbi, Alessandro Lameiras Koerich, Marco Pedersoli, Eric Granger

Abstract

Facial expression recognition (FER) in videos requires model personalization to capture the considerable variations across subjects. Vision-language models (VLMs) offer strong transfer to downstream tasks through image-text alignment, but their performance can still degrade under inter-subject distribution shifts. Personalizing models using test-time adaptation (TTA) methods can mitigate this challenge. However, most state-of-the-art TTA methods rely on unsupervised parameter optimization, introducing computational overhead that is impractical in many real-world applications. This paper introduces TTA through Cache Personalization (TTA-CaP), a cache-based TTA method that enables cost-effective (gradient-free) personalization of VLMs for video FER. Prior cache-based TTA methods rely solely on dynamic memories that store test samples, which can accumulate errors and drift due to noisy pseudo-labels. TTA-CaP leverages three coordinated caches: a personalized source cache that stores source-domain prototypes, a positive target cache that accumulates reliable subject-specific samples, and a negative target cache that stores low-confidence cases as negative samples to reduce the impact of noisy pseudo-labels. Cache updates and replacement are controlled by a tri-gate mechanism based on temporal stability, confidence, and consistency with the personalized cache. Finally, TTA-CaP refines predictions through fusion of embeddings, yielding refined representations that support temporally stable video-level predictions. Our experiments on three challenging video FER datasets, BioVid, StressID, and BAH, indicate that TTA-CaP can outperform state-of-the-art TTA methods under subject-specific and environmental shifts, while maintaining low computational and memory overhead for real-world deployment.

Test-Time Adaptation via Cache Personalization for Facial Expression Recognition in Videos

Abstract

Facial expression recognition (FER) in videos requires model personalization to capture the considerable variations across subjects. Vision-language models (VLMs) offer strong transfer to downstream tasks through image-text alignment, but their performance can still degrade under inter-subject distribution shifts. Personalizing models using test-time adaptation (TTA) methods can mitigate this challenge. However, most state-of-the-art TTA methods rely on unsupervised parameter optimization, introducing computational overhead that is impractical in many real-world applications. This paper introduces TTA through Cache Personalization (TTA-CaP), a cache-based TTA method that enables cost-effective (gradient-free) personalization of VLMs for video FER. Prior cache-based TTA methods rely solely on dynamic memories that store test samples, which can accumulate errors and drift due to noisy pseudo-labels. TTA-CaP leverages three coordinated caches: a personalized source cache that stores source-domain prototypes, a positive target cache that accumulates reliable subject-specific samples, and a negative target cache that stores low-confidence cases as negative samples to reduce the impact of noisy pseudo-labels. Cache updates and replacement are controlled by a tri-gate mechanism based on temporal stability, confidence, and consistency with the personalized cache. Finally, TTA-CaP refines predictions through fusion of embeddings, yielding refined representations that support temporally stable video-level predictions. Our experiments on three challenging video FER datasets, BioVid, StressID, and BAH, indicate that TTA-CaP can outperform state-of-the-art TTA methods under subject-specific and environmental shifts, while maintaining low computational and memory overhead for real-world deployment.
Paper Structure (14 sections, 4 equations, 5 figures, 6 tables, 1 algorithm)

This paper contains 14 sections, 4 equations, 5 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed TTA-CaP Method
  4. Personalized Source Cache
  5. Online Dynamic Target Caches
  6. Online TTA
  7. Results and Discussion
  8. Experimental Protocol
  9. Datasets.
  10. Comparison with State-of-the-Art Methods
  11. Qualitative Visualization
  12. Computation Complexity
  13. Ablation Studies
  14. Conclusion

Figures (5)

  • Figure 1: Left: Comparison of SOTA cache-based TTA methods (a) against TTA-CaP (b). TTA-CaP adapts from a test video by combining a fixed personalized cache containing source domain prototypes with dynamic target-domain caches, and refines predictions via embedding-level fusion. (c) Average weighted average recall (WAR) versus test-time training GFLOPs on the BioVid dataset. Marker shape denotes cache- vs. prompt-based methods, and marker size indicates runtime (ms).
  • Figure 2: Overview of our TTA-CaP. (top) Offline source prototype extraction. For each source-domain subject, CLIP image embeddings are clustered, and representative prototypes are selected to define a labeled set of source prototypes. (bottom) Online TTA and tri-gate cache update. At test time, frozen CLIP encoders produce text and visual embeddings and an initial prediction through contrastive similarity. A fixed personalized source cache retrieves the top-$m$ nearest source prototypes as a stable reference. A tri-gate controls updates to a dynamic target cache with positive and negative entries. Finally, predictions are obtained by embedding-level fusion of the CLIP visual embedding with retrieved cache items, enabling cost-effective personalization without training (backpropagation) at test-time.
  • Figure 3: Qualitative visualization.Left: t-SNE visualization of the embedding space before/after cache fusion for two subjects from StressID. Right: gate pass rates (fraction of frames passing each criterion and updating the caches).
  • Figure 4: Impact of selecting the top-$m$ most similar source subjects for the personalized source cache (WAR on target subjects).
  • Figure 5: (a) Average WAR for target subjects in BioVid for zero-shot CLIP, static personalized source cache only, dynamic target cache only, and TTA-CaP (static + dynamic). (b) Ablation of source-cache construction in BioVid. Average WAR across target subjects. Prototypes: the personalized source cache built from source-subject prototypes.