Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ANNA: Abstractive Text-to-Image Synthesis with Filtered News Captions

Aashish Anantha Ramakrishnan, Sharon X. Huang, Dongwon Lee

TL;DR

The paper addresses the challenge of generating images from abstractive, context-rich news captions, a setting where traditional descriptive prompts fall short. It introduces ANNA, a ~29k image-caption dataset drawn from NYTimes800K that emphasizes contextual cues and reduces overt named-entity leakage, and benchmarks state-of-the-art T2I models in zero-shot and LoRA/ReFL fine-tuning regimes. Results reveal that transfer learning yields limited gains and that improvements in image fidelity do not consistently translate to caption–image alignment, underscoring the need for models capable of deeper content-context reasoning. The work provides a valuable benchmark and analysis framework for linguistically aware T2I synthesis in the news domain, with practical implications for journalism workflows and information retrieval, while also highlighting limitations and potential societal risks associated with news image generation.

Abstract

Advancements in Text-to-Image synthesis over recent years have focused more on improving the quality of generated samples using datasets with descriptive prompts. However, real-world image-caption pairs present in domains such as news data do not use simple and directly descriptive captions. With captions containing information on both the image content and underlying contextual cues, they become abstractive in nature. In this paper, we launch ANNA, an Abstractive News captioNs dAtaset extracted from online news articles in a variety of different contexts. We explore the capabilities of current Text-to-Image synthesis models to generate news domain-specific images using abstractive captions by benchmarking them on ANNA, in both standard training and transfer learning settings. The generated images are judged on the basis of contextual relevance, visual quality, and perceptual similarity to ground-truth image-caption pairs. Through our experiments, we show that techniques such as transfer learning achieve limited success in understanding abstractive captions but still fail to consistently learn the relationships between content and context features. The Dataset is available at https://github.com/aashish2000/ANNA .

ANNA: Abstractive Text-to-Image Synthesis with Filtered News Captions

TL;DR

The paper addresses the challenge of generating images from abstractive, context-rich news captions, a setting where traditional descriptive prompts fall short. It introduces ANNA, a ~29k image-caption dataset drawn from NYTimes800K that emphasizes contextual cues and reduces overt named-entity leakage, and benchmarks state-of-the-art T2I models in zero-shot and LoRA/ReFL fine-tuning regimes. Results reveal that transfer learning yields limited gains and that improvements in image fidelity do not consistently translate to caption–image alignment, underscoring the need for models capable of deeper content-context reasoning. The work provides a valuable benchmark and analysis framework for linguistically aware T2I synthesis in the news domain, with practical implications for journalism workflows and information retrieval, while also highlighting limitations and potential societal risks associated with news image generation.

Abstract

Advancements in Text-to-Image synthesis over recent years have focused more on improving the quality of generated samples using datasets with descriptive prompts. However, real-world image-caption pairs present in domains such as news data do not use simple and directly descriptive captions. With captions containing information on both the image content and underlying contextual cues, they become abstractive in nature. In this paper, we launch ANNA, an Abstractive News captioNs dAtaset extracted from online news articles in a variety of different contexts. We explore the capabilities of current Text-to-Image synthesis models to generate news domain-specific images using abstractive captions by benchmarking them on ANNA, in both standard training and transfer learning settings. The generated images are judged on the basis of contextual relevance, visual quality, and perceptual similarity to ground-truth image-caption pairs. Through our experiments, we show that techniques such as transfer learning achieve limited success in understanding abstractive captions but still fail to consistently learn the relationships between content and context features. The Dataset is available at https://github.com/aashish2000/ANNA .
Paper Structure (16 sections, 5 figures, 2 tables)

This paper contains 16 sections, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Text-to-Image Synthesis
  4. Latent Diffusion Models
  5. Datasets
  6. Dataset Construction
  7. Pre-processing and Filtering Approaches
  8. Dataset Insights
  9. Caption Statistics
  10. News Image Analysis
  11. Categories of News Articles Selected
  12. Evaluation Methodology
  13. Experiments
  14. Quantitative Results
  15. Qualitative Results
  16. ...and 1 more sections

Figures (5)

  • Figure 1: Example of descriptive captions from the COCO Captions dataset Chen2015-qj (Left) and abstractive captions from the ANNA (Right). Words highlighted in Blue directly translate to visual entities while words highlighted in Red showcase contextual cues and syntactic variations present.
  • Figure 2: Object Frequency Analysis using Treemaps
  • Figure 3: Visualizing Article Categories of image-caption pairs present in ANNA
  • Figure 4: Qualitative comparison of different T2I models on ANNA Dataset
  • Figure :