ZeroNLG: Aligning and Autoencoding Domains for Zero-Shot Multimodal and Multilingual Natural Language Generation

TL;DR

ZeroNLG addresses the challenge of data-scarce NLG across vision, video, and multilingual text by learning a shared latent space that aligns and bridges vision and language domains through cross-domain alignment and denoising language reconstruction. The approach uses English-centric pre-training with distinct, non-overlapping corpora (WIT for vision–English; CC3M translations for En–Zh/De/Fr) and learns a multilingual auto-encoder to enable zero-shot generation, including image/video captioning and machine translation across four languages. Empirical results on twelve tasks show that ZeroNLG outperforms existing zero-shot methods in both multimodal and multilingual settings, with notable robustness and potential for extension to additional languages and modalities. This framework advances zero-shot NLG practicality for low-resource languages and lays groundwork for scalable, multilingual, multimodal data-to-text systems.

Abstract

Natural Language Generation (NLG) accepts input data in the form of images, videos, or text and generates corresponding natural language text as output. Existing NLG methods mainly adopt a supervised approach and rely heavily on coupled data-to-text pairs. However, for many targeted scenarios and for non-English languages, sufficient quantities of labeled data are often not available. To relax the dependency on labeled data of downstream tasks, we propose an intuitive and effective zero-shot learning framework, ZeroNLG, which can deal with multiple NLG tasks, including image-to-text (image captioning), video-to-text (video captioning), and text-to-text (neural machine translation), across English, Chinese, German, and French within a unified framework. ZeroNLG does not require any labeled downstream pairs for training. During training, ZeroNLG (i) projects different domains (across modalities and languages) to corresponding coordinates in a shared common latent space; (ii) bridges different domains by aligning their corresponding coordinates in this space; and (iii) builds an unsupervised multilingual auto-encoder to learn to generate text by reconstructing the input text given its coordinate in shared latent space. Consequently, during inference, based on the data-to-text pipeline, ZeroNLG can generate target sentences across different languages given the coordinate of input data in the common space. Within this unified framework, given visual (imaging or video) data as input, ZeroNLG can perform zero-shot visual captioning; given textual sentences as input, ZeroNLG can perform zero-shot machine translation. We present the results of extensive experiments on twelve NLG tasks, showing that, without using any labeled downstream pairs for training, ZeroNLG generates high-quality and believable outputs and significantly outperforms existing zero-shot methods.

Figures (5)

• Figure 1: During training, ZeroNLG first (i) projects different data across modalities and languages to corresponding coordinates in a shared common latent space; (ii) aligns their coordinates to bridge different domains; Here $S_i$ and $S_j$ refer to the text in non-English text, e.g. Chinese and Germany; (iii) performs unsupervised auto-encoding to learn to generate/reconstruct text given the coordinate of input text in this space. During inference, ZeroNLG encodes the input data acquiring its coordinate in this space, which can be directly used to perform zero-shot data-to-text generation (i.e., visual captioning and machine translation) without the need for downstream labeled pairs.
• Figure 2: The illustration of our proposed ZeroNLG, including two components: cross-domain alignment and denoising language reconstruction, where the former aims to align and bridge different data in a shared common latent space and the latter aims to reconstruct the input sentences across different languages, learning to generate sentences based on the embeddings in the common latent space. We rely on English-centric pairs for training, i.e., vision-English, English-Chinese, English-German, and English-French, where the English texts in different sets have no overlap. During inference, we can perform zero-shot natural language generation, including vision-to-Chinese/German/French captioning, and Chinese $\leftrightarrow$ German, Chinese $\leftrightarrow$ French, and German $\leftrightarrow$ French machine translation.
• Figure 3: Results of vision-to-text visual captioning with respect to different ratios of downstream data used for training. The absolute margins between our model and the state-of-the-art (SoTA) model ClipCapmokady2021clipcap are shown with the polyline. Our method consistently and significantly outperforms the SoTA under the very limited pairs setting (i.e., 0.01%, 0.1%, and 1%).
• Figure 4: We show the t-SNE visualization tsne of vision and multilingual embeddings. We plot the scatter diagrams with 200 samples for each modality and language. For comparison, we show the embeddings learned by (a) the Base model (i.e., without our CDA and DLR), (b) the Base model with CDA (Eq. \ref{['eq:formulation']}), and (c) our full model ZeroNLG.
• Figure 5: The examples of visual captions generated by the state-of-the-art zero-shot model nukrai2022text and our ZeroNLG model for different languages, i.e., English, Chinese, German, and French, under the zero-shot setting. For better understanding, we add English translations below the non-English captions in brackets. We highlight accurate keywords and wrong details. As we can see, ZeroNLG can generate accurate and vivid descriptions with more visual details across languages.