Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-LLM Collaborative Caption Generation in Scientific Documents

Jaeyoung Kim, Jongho Lee, Hong-Jun Choi, Ting-Yao Hsu, Chieh-Yang Huang, Sungchul Kim, Ryan Rossi, Tong Yu, Clyde Lee Giles, Ting-Hao 'Kenneth' Huang, Sungchul Choi

TL;DR

MLBCAP tackles the challenge of scientific figure captioning by integrating textual and visual cues through a three-stage pipeline: quality assessment, diverse caption generation, and judgment. It cleans training data from arXiv-derived captions, generates diverse candidate captions with four specialized LLMs, and selects/refines the best caption using a prominent LLM, producing both long and short versions. Human evaluations show MLBCAP captions are preferred to author-written captions and outperform baselines, though traditional metrics like BLEU/ROUGE do not reliably reflect quality. The approach demonstrates the value of multi-LLM collaboration for high-quality, multimodal figure captions in scientific documents and provides a practical framework with code available at the project repository.

Abstract

Scientific figure captioning is a complex task that requires generating contextually appropriate descriptions of visual content. However, existing methods often fall short by utilizing incomplete information, treating the task solely as either an image-to-text or text summarization problem. This limitation hinders the generation of high-quality captions that fully capture the necessary details. Moreover, existing data sourced from arXiv papers contain low-quality captions, posing significant challenges for training large language models (LLMs). In this paper, we introduce a framework called Multi-LLM Collaborative Figure Caption Generation (MLBCAP) to address these challenges by leveraging specialized LLMs for distinct sub-tasks. Our approach unfolds in three key modules: (Quality Assessment) We utilize multimodal LLMs to assess the quality of training data, enabling the filtration of low-quality captions. (Diverse Caption Generation) We then employ a strategy of fine-tuning/prompting multiple LLMs on the captioning task to generate candidate captions. (Judgment) Lastly, we prompt a prominent LLM to select the highest quality caption from the candidates, followed by refining any remaining inaccuracies. Human evaluations demonstrate that informative captions produced by our approach rank better than human-written captions, highlighting its effectiveness. Our code is available at https://github.com/teamreboott/MLBCAP

Multi-LLM Collaborative Caption Generation in Scientific Documents

TL;DR

MLBCAP tackles the challenge of scientific figure captioning by integrating textual and visual cues through a three-stage pipeline: quality assessment, diverse caption generation, and judgment. It cleans training data from arXiv-derived captions, generates diverse candidate captions with four specialized LLMs, and selects/refines the best caption using a prominent LLM, producing both long and short versions. Human evaluations show MLBCAP captions are preferred to author-written captions and outperform baselines, though traditional metrics like BLEU/ROUGE do not reliably reflect quality. The approach demonstrates the value of multi-LLM collaboration for high-quality, multimodal figure captions in scientific documents and provides a practical framework with code available at the project repository.

Abstract

Scientific figure captioning is a complex task that requires generating contextually appropriate descriptions of visual content. However, existing methods often fall short by utilizing incomplete information, treating the task solely as either an image-to-text or text summarization problem. This limitation hinders the generation of high-quality captions that fully capture the necessary details. Moreover, existing data sourced from arXiv papers contain low-quality captions, posing significant challenges for training large language models (LLMs). In this paper, we introduce a framework called Multi-LLM Collaborative Figure Caption Generation (MLBCAP) to address these challenges by leveraging specialized LLMs for distinct sub-tasks. Our approach unfolds in three key modules: (Quality Assessment) We utilize multimodal LLMs to assess the quality of training data, enabling the filtration of low-quality captions. (Diverse Caption Generation) We then employ a strategy of fine-tuning/prompting multiple LLMs on the captioning task to generate candidate captions. (Judgment) Lastly, we prompt a prominent LLM to select the highest quality caption from the candidates, followed by refining any remaining inaccuracies. Human evaluations demonstrate that informative captions produced by our approach rank better than human-written captions, highlighting its effectiveness. Our code is available at https://github.com/teamreboott/MLBCAP
Paper Structure (23 sections, 2 figures, 11 tables)

This paper contains 23 sections, 2 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Collaboration Techniques with LLMs
  4. Figure Captioning in Scientific Documents
  5. Evaluating Natural Language Generation (NLG) Tasks
  6. Problem Statement
  7. Multi-LLM Collaborative Figure Caption Generation
  8. Quality Assessment
  9. Diverse Caption Generation
  10. Judgement
  11. Experiments
  12. Dataset
  13. Training Details
  14. Human Evaluation Results
  15. Comparison with Baselines
  16. ...and 8 more sections

Figures (2)

  • Figure 1: Overview of the collaborative framework integrating multiple LLMs for caption generation in scientific documents. Initially, two MLLMs generate figure descriptions. Next, three fine-tuned models and GPT-4o generate candidate captions. Finally, GPT-4o selects and refines the best caption from the candidates.
  • Figure 2: The human evaluation results for selecting captions based on quality. Captions generated by MLBCAP (long) are most frequently selected as high quality.