Éclair -- Extracting Content and Layout with Integrated Reading Order for Documents

TL;DR

Éclair addresses the challenge of extracting richly structured text from diverse documents by learning an end-to-end model that outputs formatted text, bounding boxes, and semantic labels in reading order. It introduces arXiv-5M, a LaTeX-grounded data generation pipeline, and DROBS, a human-labeled benchmark for document-level OCR and semantic classification. The model uses a ViT-based vision encoder and a MBART-style decoder, with a flexible prompt system including a maximal-information prompt, enabling outputs that range from plain text to fully labeled content. Experimental results show state-of-the-art performance on DROBS, competitive results on existing OCR and layout benchmarks, and practical gains for downstream LLM training, with notable speedups from multi-token inference.

Abstract

Optical Character Recognition (OCR) technology is widely used to extract text from images of documents, facilitating efficient digitization and data retrieval. However, merely extracting text is insufficient when dealing with complex documents. Fully comprehending such documents requires an understanding of their structure -- including formatting, formulas, tables, and the reading order of multiple blocks and columns across multiple pages -- as well as semantic information for detecting elements like footnotes and image captions. This comprehensive understanding is crucial for downstream tasks such as retrieval, document question answering, and data curation for training Large Language Models (LLMs) and Vision Language Models (VLMs). To address this, we introduce Éclair, a general-purpose text-extraction tool specifically designed to process a wide range of document types. Given an image, Éclair is able to extract formatted text in reading order, along with bounding boxes and their corresponding semantic classes. To thoroughly evaluate these novel capabilities, we introduce our diverse human-annotated benchmark for document-level OCR and semantic classification. Éclair achieves state-of-the-art accuracy on this benchmark, outperforming other methods across key metrics. Additionally, we evaluate Éclair on established benchmarks, demonstrating its versatility and strength across several evaluation standards.

Figures (8)

• Figure 1: ÉCLAIR outperforms other methods on complex documents: (a) tables, formulas, figure, page header and multiple columns; (b) uneven columns, styling, figure; (c) non-obvious reading order and visual elements like background coloring.
• Figure 2: Meta architecture for ÉCLAIR showcasing the usage with two different (out of eight valid) prompts: Example a) uses the maximal information prompt to return bounding boxes along with their semantic class, markdown text, and tables and formulas. In b) we ask the model to return only markdown text without boxes or classes. All supported semantic classes are listed on the right.
• Figure 3: Example pages from DROBS, our visually diverse document benchmark.
• Figure S1: Illustrations of reading order over relevant text-like elements, i.e. Text, Section-header, List-item, Title and Formula. Other semantic classes (such as Picture, Footnote and Page-footer in the examples here) are not included in the reading order of the main body. (Note: We are not showing all the classes)
• Figure S2: Examples of hallucinations in the ÉCLAIR predictions. The hallucinations (in this case, repetition loops), marked in red, are detected and filtered out by our hallucination-mitigation strategy.