Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning to Hear by Seeing: It's Time for Vision Language Models to Understand Artistic Emotion from Sight and Sound

Dengming Zhang, Weitao You, Jingxiong Li, Weishen Lin, Wenda Shi, Xue Zhao, Heda Zuo, Junxian Wu, Lingyun Sun

TL;DR

The paper tackles artistic-emotion understanding by jointly leveraging sight and sound without requiring large-scale audio pretraining. It introduces VAEmotionLLM, a two-stage framework combining Vision-Guided Audio Alignment (VG-Align) to teach hearing by seeing and a Cross-Modal Emotion Adapter (EmoAdapter) to enhance cross-modal emotion understanding, evaluated on the ArtEmoBenchmark. VG-Align transfers visual reasoning into an audio pathway via logit alignment on synchronized clips, while EmoAdapter adds emotion-sensitive residuals and explicit Valence-Arousal supervision across audio, visual, and audio-visual paths; results show state-of-the-art performance across audio-only, visual-only, and joint inputs with strong ablations confirming component complementarity. The approach advances vision-language models toward reliable artistic emotion interpretation and offers practical implications for multimodal AI alignment and creative content analysis.

Abstract

Emotion understanding is critical for making Large Language Models (LLMs) more general, reliable, and aligned with humans. Art conveys emotion through the joint design of visual and auditory elements, yet most prior work is human-centered or single-modality, overlooking the emotion intentionally expressed by the artwork. Meanwhile, current Audio-Visual Language Models (AVLMs) typically require large-scale audio pretraining to endow Visual Language Models (VLMs) with hearing, which limits scalability. We present Vision Anchored Audio-Visual Emotion LLM (VAEmotionLLM), a two-stage framework that teaches a VLM to hear by seeing with limited audio pretraining and to understand emotion across modalities. In Stage 1, Vision-Guided Audio Alignment (VG-Align) distills the frozen visual pathway into a new audio pathway by aligning next-token distributions of the shared LLM on synchronized audio-video clips, enabling hearing without a large audio dataset. In Stage 2, a lightweight Cross-Modal Emotion Adapter (EmoAdapter), composed of the Emotion Enhancer and the Emotion Supervisor, injects emotion-sensitive residuals and applies emotion supervision to enhance cross-modal emotion understanding. We also construct ArtEmoBenchmark, an art-centric emotion benchmark that evaluates content and emotion understanding under audio-only, visual-only, and audio-visual inputs. VAEmotionLLM achieves state-of-the-art results on ArtEmoBenchmark, outperforming audio-only, visual-only, and audio-visual baselines. Ablations show that the proposed components are complementary.

Learning to Hear by Seeing: It's Time for Vision Language Models to Understand Artistic Emotion from Sight and Sound

TL;DR

The paper tackles artistic-emotion understanding by jointly leveraging sight and sound without requiring large-scale audio pretraining. It introduces VAEmotionLLM, a two-stage framework combining Vision-Guided Audio Alignment (VG-Align) to teach hearing by seeing and a Cross-Modal Emotion Adapter (EmoAdapter) to enhance cross-modal emotion understanding, evaluated on the ArtEmoBenchmark. VG-Align transfers visual reasoning into an audio pathway via logit alignment on synchronized clips, while EmoAdapter adds emotion-sensitive residuals and explicit Valence-Arousal supervision across audio, visual, and audio-visual paths; results show state-of-the-art performance across audio-only, visual-only, and joint inputs with strong ablations confirming component complementarity. The approach advances vision-language models toward reliable artistic emotion interpretation and offers practical implications for multimodal AI alignment and creative content analysis.

Abstract

Emotion understanding is critical for making Large Language Models (LLMs) more general, reliable, and aligned with humans. Art conveys emotion through the joint design of visual and auditory elements, yet most prior work is human-centered or single-modality, overlooking the emotion intentionally expressed by the artwork. Meanwhile, current Audio-Visual Language Models (AVLMs) typically require large-scale audio pretraining to endow Visual Language Models (VLMs) with hearing, which limits scalability. We present Vision Anchored Audio-Visual Emotion LLM (VAEmotionLLM), a two-stage framework that teaches a VLM to hear by seeing with limited audio pretraining and to understand emotion across modalities. In Stage 1, Vision-Guided Audio Alignment (VG-Align) distills the frozen visual pathway into a new audio pathway by aligning next-token distributions of the shared LLM on synchronized audio-video clips, enabling hearing without a large audio dataset. In Stage 2, a lightweight Cross-Modal Emotion Adapter (EmoAdapter), composed of the Emotion Enhancer and the Emotion Supervisor, injects emotion-sensitive residuals and applies emotion supervision to enhance cross-modal emotion understanding. We also construct ArtEmoBenchmark, an art-centric emotion benchmark that evaluates content and emotion understanding under audio-only, visual-only, and audio-visual inputs. VAEmotionLLM achieves state-of-the-art results on ArtEmoBenchmark, outperforming audio-only, visual-only, and audio-visual baselines. Ablations show that the proposed components are complementary.

Paper Structure

This paper contains 17 sections, 8 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. MLLMs: From VLMs to AVLMs
  4. Emotion Understanding Based on LLMs
  5. Method
  6. Notation and terminology.
  7. Stage 1: Let VLM Hear with Eyes
  8. Stage 2: Cross-Modal Emotion Adapter
  9. Implementation Details
  10. Experiments
  11. Benchmark and Baselines
  12. Dataset and Art-centric Emotion Benchmark
  13. Have VLMs Learned to Hear?
  14. Can LLMs Understand Artistic Emotion?
  15. What Drives the Gains?
  16. ...and 2 more sections

Figures (2)

  • Figure 1: Vision-Guided Audio Alignment. Left: training distills a frozen visual pathway (teacher) into an audio pathway (student) by matching next-token distributions under a shared prompt while the VLM and the vision encoder are frozen. Right: after training, either video or audio can be fed to the shared LLM to produce aligned responses.
  • Figure 2: Architecture of the EmoAdapter (Stage 2). The Emotion Enhancer injects lightweight residuals into audio and video tokens to form emotion-sensitive multimodal tokens. The Emotion Supervisor aggregates tokens and predicts V-A labels for three pathways (audio-only, visual-only, and audio-visual joint).