M-Longdoc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning Framework

TL;DR

This paper introduces M-LongDoc, a benchmark of 851 multimodal long documents to evaluate large models on open-ended question answering that involves text, figures, and tables. It also presents a retrieval-aware tuning framework that trains models to leverage relevant retrieved pages while resisting distractors, significantly improving performance. Empirical results show a 4.6% relative gain in correctness for open-source models, and the work includes a fully automatic training corpus to support retrieval-based tuning. The authors provide data, code, and models to facilitate ongoing research in robust multimodal long-document understanding and retrieval-grounded reasoning. Overall, M-LongDoc targets realistic, business- and research-relevant tasks and pushes toward more capable, retrieval-aware multimodal systems.

Abstract

The ability to understand and answer questions over documents can be useful in many business and practical applications. However, documents often contain lengthy and diverse multimodal contents such as texts, figures, and tables, which are very time-consuming for humans to read thoroughly. Hence, there is an urgent need to develop effective and automated methods to aid humans in this task. In this work, we introduce M-LongDoc, a benchmark of 851 samples, and an automated framework to evaluate the performance of large multimodal models. We further propose a retrieval-aware tuning approach for efficient and effective multimodal document reading. Compared to existing works, our benchmark consists of more recent and lengthy documents with hundreds of pages, while also requiring open-ended solutions and not just extractive answers. To our knowledge, our training framework is the first to directly address the retrieval setting for multimodal long documents. To enable tuning open-source models, we construct a training corpus in a fully automatic manner for the question-answering task over such documents. Experiments show that our tuning approach achieves a relative improvement of 4.6% for the correctness of model responses, compared to the baseline open-source models. Our data, code, and models are available at https://multimodal-documents.github.io.

Figures (6)

• Figure 1: Data distribution of document topics in our M-LongDoc benchmark.
• Figure 2: Comparison of benchmarks along three dimensions: the number of pages per document, the number of tokens per document, and the nature of the responses required. Specifically, we assess whether each benchmark emphasizes in-depth, comprehensive answers or focuses on short or extractive responses.
• Figure 3: Example questions in different multimodal document question answering benchmarks. For illustration, we include content from the relevant page in the original document. The example question from M-LongDoc is more complex than those from other benchmarks, as it requires an explanatory answer rather than an extraction of a short text span. Furthermore, it requires the model to understand the semantics of both image and text. Please note that in our benchmark setting, the model is provided with all page contents from the document, and not only the relevant page.
• Figure 4: Overview of our data construction process with question verification stages. For brevity, we shorten the checklist prompts and include the full details in Appendix \ref{['sec:annotation']}.
• Figure 5: Our automated evaluation framework to assess the correctness of open-ended solutions for multimodal question answering. The full evaluation guide is included in Appendix \ref{['sec:evaluation_guide']}.