Opening the Black Box: Preliminary Insights into Affective Modeling in Multimodal Foundation Models

TL;DR

Understanding where affective capabilities are encoded in multimodal foundation models, the paper conducts a mechanistic analysis across architectures and tasks and finds that affective adaptation concentrates in the FFN gating projection ($gate_proj$), not the attention pathway. Through controlled inheritance, targeted module adaptation, and ablation, it demonstrates that $gate_proj$ is sufficient, efficient, and necessary for affective understanding and generation, and introduces Gate-Focused Efficient Tuning (GET) to confine adaptation to this pathway. GET achieves about $96.6\%$ of the mean performance of AffectGPT while updating only about $24.5\%$ of the trainable parameters, highlighting substantial parameter efficiency. The work thus reveals a structural specialization separating reasoning (attention-driven) from emotion (gating-driven) and provides design principles for controllable, efficient affective capabilities in multimodal foundation models.

Abstract

Understanding where and how emotions are represented in large-scale foundation models remains an open problem, particularly in multimodal affective settings. Despite the strong empirical performance of recent affective models, the internal architectural mechanisms that support affective understanding and generation are still poorly understood. In this work, we present a systematic mechanistic study of affective modeling in multimodal foundation models. Across multiple architectures, training strategies, and affective tasks, we analyze how emotion-oriented supervision reshapes internal model parameters. Our results consistently reveal a clear and robust pattern: affective adaptation does not primarily focus on the attention module, but instead localizes to the feed-forward gating projection (\texttt{gate\_proj}). Through controlled module transfer, targeted single-module adaptation, and destructive ablation, we further demonstrate that \texttt{gate\_proj} is sufficient, efficient, and necessary for affective understanding and generation. Notably, by tuning only approximately 24.5\% of the parameters tuned by AffectGPT, our approach achieves 96.6\% of its average performance across eight affective tasks, highlighting substantial parameter efficiency. Together, these findings provide empirical evidence that affective capabilities in foundation models are structurally mediated by feed-forward gating mechanisms and identify \texttt{gate\_proj} as a central architectural locus of affective modeling.

Figures (10)

• Figure 1: Motivation of this work: opening the black box of affective modeling. This work studies the internal locus of affective modeling in foundation models and suggests that affective adaptation is closely associated with feed-forward gating mechanisms.
• Figure 2: Architectural schematic of a Transformer-based LLM. Each layer consists of a multi-head attention module with {q,k,v,o}_proj and a feed-forward network with {gate,up,down}_proj.
• Figure 3: Overall workflow of this study.
• Figure 4: Layer-wise weight changes induced by LoRA-based affective fine-tuning (relative to the base model). Across 1.5B, 7B, 14B, and 32B models, gate_proj exhibits markedly larger L2 distances than all other projection modules, revealing a consistent concentration of affective adaptation in the FFN gating mechanism.
• Figure 5: Layer-wise weight changes in LoRA-finetuned emotional and non-emotional models. Specifically, GSM8K (mathematical) fine-tuning induces pronounced weight deviations in up_proj, whereas emotional fine-tuning (AffectGPT) consistently yields the largest deviations in gate_proj, producing the most salient high-intensity band across layers.