Five Years of SciCap: What We Learned and Future Directions for Scientific Figure Captioning
Ting-Hao K. Huang, Ryan A. Rossi, Sungchul Kim, Tong Yu, Ting-Yao E. Hsu, Ho Yin, Ng, C. Lee Giles
TL;DR
SciCap investigates scientific figure captioning as a context-rich, document-grounded task rather than a pure image-to-text problem. By building the large-scale SciCap dataset and expanding through the SciCap Challenge, the work reveals that most caption content derives from figure-mentioning paragraphs and OCR, and demonstrates the utility of LLM-based evaluation and writer-assistance tools. The study advances through three threads—contextual captioning, evaluation methodology, and writer-focused interfaces (e.g., SciCapenter and multimodal profiles)—to show that while LLMs and multimodal systems can substantially aid readers and writers, caption quality remains highly audience- and context-dependent. The paper concludes with five forward-looking challenges, proposing directions to better integrate captioning with figure understanding and real-writing workflows, thereby enhancing scientific communication and multimodal language research.
Abstract
Between 2021 and 2025, the SciCap project grew from a small seed-funded idea at The Pennsylvania State University (Penn State) into one of the central efforts shaping the scientific figure-captioning landscape. Supported by a Penn State seed grant, Adobe, and the Alfred P. Sloan Foundation, what began as our attempt to test whether domain-specific training, which was successful in text models like SciBERT, could also work for figure captions expanded into a multi-institution collaboration. Over these five years, we curated, released, and continually updated a large collection of figure-caption pairs from arXiv papers, conducted extensive automatic and human evaluations on both generated and author-written captions, navigated the rapid rise of large language models (LLMs), launched annual challenges, and built interactive systems that help scientists write better captions. In this piece, we look back at the first five years of SciCap and summarize the key technical and methodological lessons we learned. We then outline five major unsolved challenges and propose directions for the next phase of research in scientific figure captioning.