LAMPRET: Layout-Aware Multimodal PreTraining for Document Understanding

TL;DR

LAMPreT presents a hierarchical, layout-aware multimodal pretraining framework that jointly models document content blocks and their spatial structure to improve document understanding. By combining a lower-level multimodal transformer with a higher-level structure-focused transformer and introducing novel pretraining objectives (block-ordering, block-MLM, image fitting), it demonstrates improvements on text block filling and near-perfect performance on image content suggestion. The approach highlights the value of explicitly incorporating layout multimodality and structure, offering a generalizable pathway for multimodal document understanding beyond templated or OCR-centric settings. The work also opens avenues for curriculum-style training and broader domain applicability, such as news articles.

Abstract

Document layout comprises both structural and visual (eg. font-sizes) information that is vital but often ignored by machine learning models. The few existing models which do use layout information only consider textual contents, and overlook the existence of contents in other modalities such as images. Additionally, spatial interactions of presented contents in a layout were never really fully exploited. To bridge this gap, we parse a document into content blocks (eg. text, table, image) and propose a novel layout-aware multimodal hierarchical framework, LAMPreT, to model the blocks and the whole document. Our LAMPreT encodes each block with a multimodal transformer in the lower-level and aggregates the block-level representations and connections utilizing a specifically designed transformer at the higher-level. We design hierarchical pretraining objectives where the lower-level model is trained similarly to multimodal grounding models, and the higher-level model is trained with our proposed novel layout-aware objectives. We evaluate the proposed model on two layout-aware tasks -- text block filling and image suggestion and show the effectiveness of our proposed hierarchical architecture as well as pretraining techniques.

Figures (4)

• Figure 1: An example page parsed by the document tokenizer. Each red box indicates a content block. Blue colored coordinates are an exemplar block position tuple (origin of the coordinate system is at top-left corner of the entire document).
• Figure 2: LAMPreT Framework Overview:(a) Sorting and serializing blocks: Setting the origin of the document at the top-left-most corner, we perform a 2D sorting on the content blocks anchoring around their top-left coordinate. We then serialize the sorted blocks in a zigzag fashion to obtain a reasonable ordering for the model inputs. (b) Hierarchical formulation:LAMPreT framework exploits the inherent hierarchical nature of document layouts. The input representation of each block $\text{blk}_{i}$ contains the embeddings of Wordpiece tokens, block-segment-ids, modalities, and attributional features. The output representations of the lower-level model at each $\text{CLS}_{i}$ position are fed to the higher-level model. Different levels of objectives are applied to the models in different hierarchy, and the representations at global-CLS are used in downstream tasks.
• Figure 3: LAMPreT Pretraining Objectives: HObj$_{i}$ and LObj$_{i}$ denotes the $i$-th high- and low-level objective respectively. MLM and ITM stands for masked-language modeling and the image-text matching prediction for the low-level objectives. For each high-level objective, we illustrate: an exemplar block swapping for the block-ordering objective, an image masking for the image fitting objective, and a block masked at its block-level representation for the block-MLM objective, respectively.
• Figure 4: Illustration of CNN-Grid baseline: The block-level representations $\text{blkh}_{i}$ are inserted to the corresponding 2D positions of a 2D map, e.g. $\text{blkh}_{21}$ of $\text{blk}_{21}$ is inserted to the position (2, 1) (X-axis is vertical and Y-axis is horizontal). This will result in a 3D tensor since as the dimension of $\text{blkh}_{i}$ will be the channel dimension of the inserted map. This 3D tensor is then fed to the convolutional modules, where the CNN module (represented as the rectangular shape) is designed to output the same shape of the input 3D tensor. Notice that the global-CLS is not added since we do not need it for CNN higher-level model, instead, we perform an average pooling for reducing the height and width of the output tensor to $1\times1$ for obtaining a 1D document-level representation. The high-level objectives can be applied to the combination of the document-level representation and each of the $\text{out}_{i}$ after serializing the output tensor of the first stage CNN module back.