Aligning Video Models with Human Social Judgments via Behavior-Guided Fine-Tuning

TL;DR

This work examines whether modern video and language models encode human-perceived similarity in social videos and demonstrates a method to align video representations with human judgments. A large dataset of 49,484 odd-one-out judgments on 250 social clips reveals a modality gap, with language embeddings outperforming video models. The authors introduce a behavior-guided fine-tuning approach using a hybrid triplet-RSA loss implemented via LoRA on TimeSformer, yielding substantial gains in alignment (higher $R^2$ and OOO accuracy) and broader semantic encoding, including social-affective attributes, without harming action recognition. The findings show that incorporating human relational structure into training can produce more human-like representations, with potential benefits for retrieval, interpretability, and social understanding in AI systems.

Abstract

Humans intuitively perceive complex social signals in visual scenes, yet it remains unclear whether state-of-the-art AI models encode the same similarity structure. We study (Q1) whether modern video and language models capture human-perceived similarity in social videos, and (Q2) how to instill this structure into models using human behavioral data. To address this, we introduce a new benchmark of over 49,000 odd-one-out similarity judgments on 250 three-second video clips of social interactions, and discover a modality gap: despite the task being visual, caption-based language embeddings align better with human similarity than any pretrained video model. We close this gap by fine-tuning a TimeSformer video model on these human judgments with our novel hybrid triplet-RSA objective using low-rank adaptation (LoRA), aligning pairwise distances to human similarity. This fine-tuning protocol yields significantly improved alignment with human perceptions on held-out videos in terms of both explained variance and odd-one-out triplet accuracy. Variance partitioning shows that the fine-tuned video model increases shared variance with language embeddings and explains additional unique variance not captured by the language model. Finally, we test transfer via linear probes and find that human-similarity fine-tuning strengthens the encoding of social-affective attributes (intimacy, valence, dominance, communication) relative to the pretrained baseline. Overall, our findings highlight a gap in pretrained video models' social recognition and demonstrate that behavior-guided fine-tuning shapes video representations toward human social perception.

Figures (4)

• Figure 1: Triplet Odd-One-Out Dataset and TimeSformer Hybrid fine-tuning. We collect similarity judgments via a triplet odd-one-out task. Human choices are used as positive and negative signals for each training loss. At every optimizer step, the model is updated with a triplet loss (blue) on a batch of Anchor (A), Positive (P), Negative (N) triplets. Periodically ($\approx 6$ times per epoch), an additional RSA loss (purple) is applied on a small subset of 24 videos with 6 as gradients by aligning the model’s pairwise distances with the human similarity derived from all triplets. The combined training objective of triplet and RSA loss is defined in Eq. \ref{['eq:hybridloss']}.
• Figure 2: Explained variance $(R^2)$ computed as Spearman’s rank correlation between model embeddings and human similarity judgments and we report its square as a measure of explained variance (differing from regression). Language models outperform pretrained video models, but fine-tuned TimeSformer exceeds both. Horizontal dashed line shows the split-half spearman correlation$^2$ of the human RSM used as our noise ceiling (Appendix §\ref{['app:splithalf']}).
• Figure 3: Variance partitioning before and after fine-tuning. Fine-tuning increases the unique variance explained by the TimeSformer (cyan), reduces the unique contribution of the language model (purple), and expands shared variance (gray). This shows that the fine-tuned video model both captures variance previously available only from captions and better overlaps with language-based structure. Total variance explained (black markers) approaches the reliability ceiling. Thicker black outline shows total variance explained by the video model.
• Figure 4: Pearson correlation ($r$) scores for predicting social-affective attributes from video embeddings using Ridge Regression. Language (purple) is the best performing language model for comparison to baseline (dark blue) and finetuned (light blue) TimeSformer.