CUTIE: Learning to Understand Documents with Convolutional Universal Text Information Extractor

TL;DR

The paper tackles robust key information extraction from diverse, template-free receipts and invoices (SROIE) by introducing grid-based text representations that fuse semantic embeddings with spatial layout. It proposes CUTIE, a CNN-based framework with two architectures (CUTIE-A and CUTIE-B) that perform end-to-end grid labeling without pretraining or post-processing, achieving state-of-the-art results on ICDAR 2019 SROIE and strong data efficiency on a large self-built dataset. The key contributions include grid positional mapping to preserve spatial relationships, multi-scale context capture via HRNet-like fusion (CUTIE-A) and atrous/ASPP-based planning (CUTIE-B), and comprehensive ablations demonstrating the benefits of grid augmentation, embedding size, and training data quantity. Practically, CUTIE offers fast, template-free extraction of structured key information from receipts with reduced data requirements, suitable for scalable document automation.

Abstract

Extracting key information from documents, such as receipts or invoices, and preserving the interested texts to structured data is crucial in the document-intensive streamline processes of office automation in areas that includes but not limited to accounting, financial, and taxation areas. To avoid designing expert rules for each specific type of document, some published works attempt to tackle the problem by learning a model to explore the semantic context in text sequences based on the Named Entity Recognition (NER) method in the NLP field. In this paper, we propose to harness the effective information from both semantic meaning and spatial distribution of texts in documents. Specifically, our proposed model, Convolutional Universal Text Information Extractor (CUTIE), applies convolutional neural networks on gridded texts where texts are embedded as features with semantical connotations. We further explore the effect of employing different structures of convolutional neural network and propose a fast and portable structure. We demonstrate the effectiveness of the proposed method on a dataset with up to $4,484$ labelled receipts, without any pre-training or post-processing, achieving state of the art performance that is much better than the NER based methods in terms of either speed and accuracy. Experimental results also demonstrate that the proposed CUTIE model being able to achieve good performance with a much smaller amount of training data.

Figures (7)

• Figure 1: Framework of the proposed method, (a) positional map the scanned document image to a grid with text's relative spatial relation preserved, (b) feed the generated grid into the CNN for extracting key information, (c) reverse map the extracted key information for visual reference.
• Figure 2: Example of scanned taxi receipt images. We provide two colored rectangles to help readers find the key information about distance of travel and total amount with blue and red, respectively. Note the different types of spatial layouts and key information texts in these receipt images.
• Figure 3: Example of CUITE inference results. Color legend in the top-left corner indicates the key information classes. Each color indicates a key information class, where filled rectangles are the ground truths while the boundary-only rectangles are the inference results. The result is perfectly correct as if the filled rectangles overlap with the boundary-only rectangles. We mask out certain private information with filled gray rectangles in these figures. (zooming in to check the details)
• Figure 4: False positive examples of CUITE prediction results. Color legend in the top-left corner indicates the key information classes. Each color indicates a key information class, where filled rectangles are the ground truths while the boundary-only rectangles are the inference results. The false positives are results with the boundary-only rectangles being not overlapped with the filled rectangles. We mask out certain private information with filled gray rectangles in these figures. (zooming in to check the details)
• Figure 5: False labelling examples of CUITE prediction results, where the error is actually caused by the wrong labelling of human labeler. Color legend in the top-left corner indicates the key information classes. Each color indicates a key information class, where filled rectangles are the ground truths while the boundary-only rectangles are the inference results. We mask out certain private information with filled gray rectangles in these figures. (zooming in to check the details)