Semi-supervised Chinese Poem-to-Painting Generation via Cycle-consistent Adversarial Networks

TL;DR

A semi-supervised approach using cycle-consistent adversarial networks to leverage the limited paired data and large unpaired corpus of poems and paintings to leverage bidirectional mappings that enforce semantic alignment between the visual and textual modalities.

Abstract

Classical Chinese poetry and painting represent the epitome of artistic expression, but the abstract and symbolic nature of their relationship poses a significant challenge for computational translation. Most existing methods rely on large-scale paired datasets, which are scarce in this domain. In this work, we propose a semi-supervised approach using cycle-consistent adversarial networks to leverage the limited paired data and large unpaired corpus of poems and paintings. The key insight is to learn bidirectional mappings that enforce semantic alignment between the visual and textual modalities. We introduce novel evaluation metrics to assess the quality, diversity, and consistency of the generated poems and paintings. Extensive experiments are conducted on a new Chinese Painting Description Dataset (CPDD). The proposed model outperforms previous methods, showing promise in capturing the symbolic essence of artistic expression. Codes are available online \url{https://github.com/Mnster00/poemtopainting}.

Figures (9)

• Figure 1: The framework of the proposed semi-supervised framework.
• Figure 2: Painting Categories Across Historical Periods
• Figure 3: Examples of pairwise poems and paintings from CPDD dataset. English descriptions are literal translations of the original Chinese poems.
• Figure 4: The framework of the proposed cycle-consistent adversarial network for Chinese poem-to-painting translation. It consists of poem and painting encoders ($E_p$, $E_t$) that map into a shared latent space, corresponding generators ($G_p$, $G_t$) that decode from this space, and discriminator ($D_p$, $D_t$) that evaluate accuracy.The encoders and generators are trained with both adversarial losses ($L_{adv}$) and cycle consistency losses ($L_{cyc}$).
• Figure 5: Cycle-consistent networks aim to address bi-directional models fitting between reciprocal tasks.