A Preliminary Exploration with GPT-4o Voice Mode

TL;DR

The paper presents a preliminary, end-to-end evaluation of GPT-4o's audio understanding and reasoning using Dynamic-SUPERB, MMAU, and CMM benchmarks across audio, speech, and music. It contrasts end-to-end LALMs with cascaded approaches and documents how post-training safeguards shape refusals and task feasibility. Key findings show GPT-4o strong multilingual ASR, speech and singing analysis capabilities, but notable weaknesses in certain audio-duration and instrument-classification tasks, as well as substantial safety-related refusals that complicate full capability assessment. The work highlights both the progress toward universal instruction-based speech models and the ongoing challenges of safety, dataset sensitivity, and evaluation protocol effects on measured performance.

Abstract

With the rise of multimodal large language models, GPT-4o stands out as a pioneering model, driving us to evaluate its capabilities. This report assesses GPT-4o across various tasks to analyze its audio processing and reasoning abilities. We find that GPT-4o exhibits strong knowledge in audio, speech, and music understanding, performing well in tasks like intent classification, spoken command classification, semantic and grammatical reasoning., multilingual speech recognition, and singing analysis. It also shows greater robustness against hallucinations than other large audio-language models (LALMs). However, it struggles with tasks such as audio duration prediction and instrument classification. Additionally, GPT-4o's safety mechanisms cause it to decline tasks like speaker identification, age classification, MOS prediction, and audio deepfake detection. Notably, the model exhibits a significantly different refusal rate when responding to speaker verification tasks on different datasets. This is likely due to variations in the accompanying instructions or the quality of the input audio, suggesting the sensitivity of its built-in safeguards. Finally, we acknowledge that model performance varies with evaluation protocols. This report only serves as a preliminary exploration of the current state of LALMs.

Figures (32)

• Figure 1: Relative performance comparison of models in the (Speech) Paralinguistics domain.
• Figure 2: Refusal rates of tasks in the (Speech) Paralinguistics domain
• Figure 3: Relative performance comparison of models in the (Speech) Phonetics, Phonology, Prosody domain.
• Figure 4: Refusal rates of tasks in the (Speech) Phonetics, Phonology, Prosody domain. The abbreviation "PFC" in the plot refers to "Phonological Feature Classification"
• Figure 5: Relative performance comparison of models in the (Speech) Safety & Security domain.