Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bridging Ears and Eyes: Analyzing Audio and Visual Large Language Models to Humans in Visible Sound Recognition and Reducing Their Sensory Gap via Cross-Modal Distillation

Xilin Jiang, Junkai Wu, Vishal Choudhari, Nima Mesgarani

TL;DR

The paper investigates modality-specific strengths of audio and visual LLMs in visible sound recognition and compares them to human perception on the same acoustic scenes. It reveals a modality-specific sensory gap that mirrors the human ear–eye divide and proposes a heuristic-guided cross-modal distillation to bridge it. The approach transfers knowledge between audio and visual LLMs, yielding substantial gains (e.g., Qwen2-Audio reaching about 92% on analysis data and approaching the multimodal Omni) and generalizing to unseen classes such as those in AudioSet. The results offer a scalable path to align modality-specific perception in multimodal LLMs and reduce sensory blind spots.

Abstract

Audio large language models (LLMs) are considered experts at recognizing sound objects, yet their performance relative to LLMs in other sensory modalities, such as visual or audio-visual LLMs, and to humans using their ears, eyes, or both remains unexplored. To investigate this, we systematically evaluate audio, visual, and audio-visual LLMs, specifically Qwen2-Audio, Qwen2-VL, and Qwen2.5-Omni, against humans in recognizing sound objects of different classes from audio-only, silent video, or sounded video inputs. We uncover a performance gap between Qwen2-Audio and Qwen2-VL that parallels the sensory discrepancy between human ears and eyes. To reduce this gap, we introduce a cross-modal distillation framework, where an LLM in one modality serves as the teacher and another as the student, with knowledge transfer in sound classes predicted as more challenging to the student by a heuristic model. Distillation in both directions, from Qwen2-VL to Qwen2-Audio and vice versa, leads to notable improvements, particularly in challenging classes. This work highlights the sensory gap in LLMs from a human-aligned perspective and proposes a principled approach to enhancing modality-specific perception in multimodal LLMs.

Bridging Ears and Eyes: Analyzing Audio and Visual Large Language Models to Humans in Visible Sound Recognition and Reducing Their Sensory Gap via Cross-Modal Distillation

TL;DR

The paper investigates modality-specific strengths of audio and visual LLMs in visible sound recognition and compares them to human perception on the same acoustic scenes. It reveals a modality-specific sensory gap that mirrors the human ear–eye divide and proposes a heuristic-guided cross-modal distillation to bridge it. The approach transfers knowledge between audio and visual LLMs, yielding substantial gains (e.g., Qwen2-Audio reaching about 92% on analysis data and approaching the multimodal Omni) and generalizing to unseen classes such as those in AudioSet. The results offer a scalable path to align modality-specific perception in multimodal LLMs and reduce sensory blind spots.

Abstract

Audio large language models (LLMs) are considered experts at recognizing sound objects, yet their performance relative to LLMs in other sensory modalities, such as visual or audio-visual LLMs, and to humans using their ears, eyes, or both remains unexplored. To investigate this, we systematically evaluate audio, visual, and audio-visual LLMs, specifically Qwen2-Audio, Qwen2-VL, and Qwen2.5-Omni, against humans in recognizing sound objects of different classes from audio-only, silent video, or sounded video inputs. We uncover a performance gap between Qwen2-Audio and Qwen2-VL that parallels the sensory discrepancy between human ears and eyes. To reduce this gap, we introduce a cross-modal distillation framework, where an LLM in one modality serves as the teacher and another as the student, with knowledge transfer in sound classes predicted as more challenging to the student by a heuristic model. Distillation in both directions, from Qwen2-VL to Qwen2-Audio and vice versa, leads to notable improvements, particularly in challenging classes. This work highlights the sensory gap in LLMs from a human-aligned perspective and proposes a principled approach to enhancing modality-specific perception in multimodal LLMs.
Paper Structure (11 sections, 3 figures, 2 tables)

This paper contains 11 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Cross-Modal Analysis
  4. Multimodal LLMs and Data
  5. Performance Analysis of Original LLMs
  6. Cross-modal Distillation
  7. Two Modalities, Same Acoustic Scene
  8. Heuristic Switch
  9. Performance Analysis of Distilled LLMs
  10. Discussion and Limitation
  11. Conclusion

Figures (3)

  • Figure 1: We compared Qwen2-Audio, Qwen2-VL, and Qwen2.5-Omni among themselves and against humans on 30 classes $\times$ 50 samples/class from VGGSound analysis set under three input conditions of the same acoustic scene: audio-only, silent video, and sounded video. Class-wise analysis reveals that the sensory gap among LLMs (audio vs. visual vs. both) mirrors human perception differences between ears, eyes, and their integration. The overall accuracy of 30 classes shown in this figure is: Qwen2.5-Omni (94.4%) $>$ Human Ears&Eyes (84.2%) $>$ Qwen2-VL (83.9%) $>$ Human Eyes (74.1%) $>$ Qwen2-Audio (72.5%) $>$ Human Ears (71.9%). Despite slightly higher overall accuracy, Qwen2-Audio underperforms human listeners in most challenging sound classes (bottom row).
  • Figure 2: Proposed cross-modal distillation framework between an audio and a visual LLM. Both LLMs perceive the same acoustic scene but in either audio or video-only format. A heuristic switch predicts whether the student (audio in this figure) LLM underperforms the teacher (visual) LLM for each sample. Based on this decision, the student LLM is finetuned either by the teacher LLM’s output or its own output.
  • Figure 3: Class-wise comparison of Qwen2-Audio before and after distillation. Top two rows: The distilled Qwen2-Audio shows substantial accuracy gains on previously challenging classes, comparable to and sometimes surpassing the Qwen2-VL. Bottom two rows: Cross-modal distillation also improves recognizing unseen classes in AudioSet.