XL-HeadTags: Leveraging Multimodal Retrieval Augmentation for the Multilingual Generation of News Headlines and Tags

TL;DR

This work tackles multilingual news headline and tag generation by introducing XL-HeadTags, a large multimodal dataset spanning 20 languages, and a retrieval-augmented generation framework (MultiRAGen) that leverages images and captions to select salient article content. The approach combines two retrieval modules (ImgRet and CapRet) with instruction-tuned generation to produce headlines and tag words in both controlled and unrestricted modes, enabling flexible generation in a multilingual setting. The authors also provide a suite of multilingual tools (ROUGE scorer, sentence tokenizer, stemmer) to support evaluation across languages. Empirical results show that multimodal retrieval improves headline and tag-word quality across languages, with caption-based retrieval often yielding stronger gains, while highlighting challenges for low-resource languages and issues with some current LLMs’ multilingual behavior. The work also discusses dataset bias, content-alignment limitations, and computational considerations, offering a practical pathway toward more effective and accessible multilingual content generation tooling and evaluation resources.

Abstract

Millions of news articles published online daily can overwhelm readers. Headlines and entity (topic) tags are essential for guiding readers to decide if the content is worth their time. While headline generation has been extensively studied, tag generation remains largely unexplored, yet it offers readers better access to topics of interest. The need for conciseness in capturing readers' attention necessitates improved content selection strategies for identifying salient and relevant segments within lengthy articles, thereby guiding language models effectively. To address this, we propose to leverage auxiliary information such as images and captions embedded in the articles to retrieve relevant sentences and utilize instruction tuning with variations to generate both headlines and tags for news articles in a multilingual context. To make use of the auxiliary information, we have compiled a dataset named XL-HeadTags, which includes 20 languages across 6 diverse language families. Through extensive evaluation, we demonstrate the effectiveness of our plug-and-play multimodal-multilingual retrievers for both tasks. Additionally, we have developed a suite of tools for processing and evaluating multilingual texts, significantly contributing to the research community by enabling more accurate and efficient analysis across languages.

Figures (3)

• Figure 1: Our content selection approach. Auxiliary information, including images and captions, is used as queries to extract salient and relevant sentences from documents via two modules: ImgRet for images (visual modality) and CapRet for image captions (textual modality). Our modules are designed as plug-and-play components that can be integrated with language models of any size and type.
• Figure 2: Distribution of Datasets and Models Across Languages. Data were sourced from the Huggingface resource ranking (https://huggingface.co/languages) as of February 5, 2024.
• Figure 3: Large Language Models (LLMs) generated samples.