μgat: Improving Single-Page Document Parsing by Providing Multi-Page Context

TL;DR

This work addresses the challenge of parsing visually rich, multi-page documents by extending single-page, OCR-free parsers to leverage surrounding page context. The authors introduce μgat, an adapter-enhanced version of Nougat, which ingests embeddings from the previous, current, and next pages to generate compact context tokens that guide the decoder in producing structured Markdown/LaTeX markup. They validate μgat on arXiv academic papers, synthetic long tables, and Regesta Pontificum Romanorum pages, showing consistent improvements over baselines and highlighting the importance of cross-page context for parsing accuracy. The approach enables robust multi-page parsing without full re-training of the encoder, offering a scalable path for VrDU tasks and Digital Humanities applications, particularly for regesta collections where layout and inter-page structure carry essential information.

Abstract

Regesta are catalogs of summaries of other documents and, in some cases, are the only source of information about the content of such full-length documents. For this reason, they are of great interest to scholars in many social and humanities fields. In this work, we focus on Regesta Pontificum Romanum, a large collection of papal registers. Regesta are visually rich documents, where the layout is as important as the text content to convey the contained information through the structure, and are inherently multi-page documents. Among Digital Humanities techniques that can help scholars efficiently exploit regesta and other documental sources in the form of scanned documents, Document Parsing has emerged as a task to process document images and convert them into machine-readable structured representations, usually markup language. However, current models focus on scientific and business documents, and most of them consider only single-paged documents. To overcome this limitation, in this work, we propose μgat, an extension of the recently proposed Document parsing Nougat architecture, which can handle elements spanning over the single page limits. Specifically, we adapt Nougat to process a larger, multi-page context, consisting of the previous and the following page, while parsing the current page. Experimental results, both qualitative and quantitative, demonstrate the effectiveness of our proposed approach also in the case of the challenging Regesta Pontificum Romanorum.

Figures (4)

• Figure 1: Three consecutive pages from the Regesta Pontificum Romanorum: summaries are both simple-layout paragraphs and text in a tabular layout, and the corresponding information about date and place is given via layout. Such entries (summaries and metadata) span multiple pages, with interleaved information and layout elements.
• Figure 2: Our proposed $\mu$gat architecture. The Visual Encoder embeds the previous, current, and next page. These embeddings are then concatenated and, after the addition of inter-page and inter-page positional embeddings, fed into the key and value of the adapter's cross-attention layers, which extract information into the learned $N$ tokens representing compressed information about the pages. These tokens are then appended to the current page tokens and fed to the auto-regressive Decoder.
• Figure 3: Qualitative results on two pages from arXiv, when no context is available (left) and full context is available (right). We report the Ground Truth and highlight the errors made by $\mu$gat and Nougat-S* in their LaTeX-rendered output.
• Figure 4: Qualitative results on two pages from the challenging RPR. We report the Ground Truth and highlight the errors made by $\mu$gat in its LaTeX-rendered output.