Enhancing Quranic Learning: A Multimodal Deep Learning Approach for Arabic Phoneme Recognition

TL;DR

The paper tackles phoneme-level mispronunciation detection in Qur’anic Arabic by introducing a transformer-based multimodal framework that fuses acoustic embeddings from UniSpeech with textual embeddings derived from Whisper transcriptions via BERT. It systematically compares early, intermediate, and late fusion strategies using two Arabic phoneme datasets, demonstrating that multimodal integration yields superior accuracy and F1-scores over a unimodal SOTA baseline. Key contributions include a dedicated Arabic phoneme dataset, a rigorous fusion strategy comparison, and a demonstration that cross-modal representations enhance pronunciation assessment for CALL tools. The work has practical significance for scalable, speaker-independent Quranic pronunciation training and lays a foundation for broader multimodal speech learning in morphologically rich languages.

Abstract

Recent advances in multimodal deep learning have greatly enhanced the capability of systems for speech analysis and pronunciation assessment. Accurate pronunciation detection remains a key challenge in Arabic, particularly in the context of Quranic recitation, where subtle phonetic differences can alter meaning. Addressing this challenge, the present study proposes a transformer-based multimodal framework for Arabic phoneme mispronunciation detection that combines acoustic and textual representations to achieve higher precision and robustness. The framework integrates UniSpeech-derived acoustic embeddings with BERT-based textual embeddings extracted from Whisper transcriptions, creating a unified representation that captures both phonetic detail and linguistic context. To determine the most effective integration strategy, early, intermediate, and late fusion methods were implemented and evaluated on two datasets containing 29 Arabic phonemes, including eight hafiz sounds, articulated by 11 native speakers. Additional speech samples collected from publicly available YouTube recordings were incorporated to enhance data diversity and generalization. Model performance was assessed using standard evaluation metrics: accuracy, precision, recall, and F1-score, allowing a detailed comparison of the fusion strategies. Experimental findings show that the UniSpeech-BERT multimodal configuration provides strong results and that fusion-based transformer architectures are effective for phoneme-level mispronunciation detection. The study contributes to the development of intelligent, speaker-independent, and multimodal Computer-Aided Language Learning (CALL) systems, offering a practical step toward technology-supported Quranic pronunciation training and broader speech-based educational applications.

Figures (2)

• Figure 1: The block diagram of proposed method
• Figure 2: The proposed deep learning-based fusion models (a) Early fusion, (b) Intermediate fusion, and (c) Late fusion.