Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Figuring out Figures: Using Textual References to Caption Scientific Figures

Stanley Cao, Kevin Liu

TL;DR

The paper tackles automatic captioning of scientific figures by integrating textual metadata (title, abstract, references) with image features in a transformer-based encoder-decoder. It introduces MetaSciCap, augmenting SciCap with targeted metadata, and demonstrates that text-driven models (SciBERT+GPT-2) can outperform image-centric variants, especially for original captions, highlighting the pivotal role of textual context. The findings suggest that while multimodal fusion helps, stronger image encoding and metadata-aware preprocessing are essential for robust scientific figure captioning, with implications for accessibility and automated literature understanding. Overall, the work advances figure captioning by quantifying the value of textual references and proposing a scalable framework for combining image and scholarly text signals.

Abstract

Figures are essential channels for densely communicating complex ideas in scientific papers. Previous work in automatically generating figure captions has been largely unsuccessful and has defaulted to using single-layer LSTMs, which no longer achieve state-of-the-art performance. In our work, we use the SciCap datasets curated by Hsu et al. and use a variant of a CLIP+GPT-2 encoder-decoder model with cross-attention to generate captions conditioned on the image. Furthermore, we augment our training pipeline by creating a new dataset MetaSciCap that incorporates textual metadata from the original paper relevant to the figure, such as the title, abstract, and in-text references. We use SciBERT to encode the textual metadata and use this encoding alongside the figure embedding. In our experimentation with different models, we found that the CLIP+GPT-2 model performs better when it receives all textual metadata from the SciBERT encoder in addition to the figure, but employing a SciBERT+GPT2 model that uses only the textual metadata achieved optimal performance.

Figuring out Figures: Using Textual References to Caption Scientific Figures

TL;DR

The paper tackles automatic captioning of scientific figures by integrating textual metadata (title, abstract, references) with image features in a transformer-based encoder-decoder. It introduces MetaSciCap, augmenting SciCap with targeted metadata, and demonstrates that text-driven models (SciBERT+GPT-2) can outperform image-centric variants, especially for original captions, highlighting the pivotal role of textual context. The findings suggest that while multimodal fusion helps, stronger image encoding and metadata-aware preprocessing are essential for robust scientific figure captioning, with implications for accessibility and automated literature understanding. Overall, the work advances figure captioning by quantifying the value of textual references and proposing a scalable framework for combining image and scholarly text signals.

Abstract

Figures are essential channels for densely communicating complex ideas in scientific papers. Previous work in automatically generating figure captions has been largely unsuccessful and has defaulted to using single-layer LSTMs, which no longer achieve state-of-the-art performance. In our work, we use the SciCap datasets curated by Hsu et al. and use a variant of a CLIP+GPT-2 encoder-decoder model with cross-attention to generate captions conditioned on the image. Furthermore, we augment our training pipeline by creating a new dataset MetaSciCap that incorporates textual metadata from the original paper relevant to the figure, such as the title, abstract, and in-text references. We use SciBERT to encode the textual metadata and use this encoding alongside the figure embedding. In our experimentation with different models, we found that the CLIP+GPT-2 model performs better when it receives all textual metadata from the SciBERT encoder in addition to the figure, but employing a SciBERT+GPT2 model that uses only the textual metadata achieved optimal performance.
Paper Structure (13 sections, 5 figures, 1 table)

This paper contains 13 sections, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Approach
  4. Experiments
  5. Data
  6. Evaluation Method
  7. Experimental Details
  8. Results
  9. Analysis
  10. Normalized Captions vs. Original Captions
  11. Performance Disparities from Modal Ablations
  12. Conclusion
  13. Appendix

Figures (5)

  • Figure 1: "Change in BLEU and ROUGE scores with the number of source predictions." This caption was actually autogenerated by our text-only model given our title, abstract, and reference. Best of 3 generations selected. In reality, this figure shows the improvement in BLEU and ROUGE over training.
  • Figure 2: Model Predictions on Normalized Captions.
  • Figure 3: Model Predictions on Original Captions
  • Figure 4: Exemplary generation from text-only model showing the potential for memorization. If very similar wording is used in both the references and gold caption, the model may achieve strikingly good performance by simply regurgitating or slightly rewording reference text.
  • Figure 5: Our model learns the exact acronyms used in figures.