Picturized and Recited with Dialects: A Multimodal Chinese Representation Framework for Sentiment Analysis of Classical Chinese Poetry

TL;DR

This work tackles sentiment analysis for classical Chinese poetry by introducing a dialect-aware tri-modal framework that jointly leverages sentence-level audio (including multiple dialects), visual features from a text-to-image model, and textual representations enhanced by LLM-generated modern translations. A two-stage training regime using multimodal contrastive learning aligns the modalities, with dialect cross-attention refining phonetic fusion. Empirical results on the THU FSPC and CCPD datasets show state-of-the-art performance, with notable gains from audio, vision, and dialect components, and the approach improves other textual backbones when integrated. The framework advances general multimodal Chinese representation and offers practical insights for broader applications in poetry understanding and beyond, with open-source code to foster further research.

Abstract

Classical Chinese poetry is a vital and enduring part of Chinese literature, conveying profound emotional resonance. Existing studies analyze sentiment based on textual meanings, overlooking the unique rhythmic and visual features inherent in poetry,especially since it is often recited and accompanied by Chinese paintings. In this work, we propose a dialect-enhanced multimodal framework for classical Chinese poetry sentiment analysis. We extract sentence-level audio features from the poetry and incorporate audio from multiple dialects,which may retain regional ancient Chinese phonetic features, enriching the phonetic representation. Additionally, we generate sentence-level visual features, and the multimodal features are fused with textual features enhanced by LLM translation through multimodal contrastive representation learning. Our framework outperforms state-of-the-art methods on two public datasets, achieving at least 2.51% improvement in accuracy and 1.63% in macro F1. We open-source the code to facilitate research in this area and provide insights for general multimodal Chinese representation.

Figures (3)

• Figure 1: Case (a) illustrates the rhythmic features of classical Chinese poetry, Case (b) shows differences in dialectal pronunciations, and Case (c) demonstrates the visual features.
• Figure 2: Tri-modal framework with textual (orange), audio (green), and visual (blue) modality modules. PLM, Transformer, Cross-Attention, CNN, and Classifier are trainable. In the first pre-training phase (the purple line), we use contrastive loss $\mathcal{L}_{\text{contrastive}}$ to train the three modality feature extractors. In the second phase, all trainable modules are jointly trained on specific downstream tasks.
• Figure 3: Comparison of model-mandarin (in (a), trained on Mandarin audio) and model-cantonese (in (b), trained on Cantonese audio) tested on poems from various provinces. The red line represents the Yangtze River, which divides northern and southern China.