Improving Applicability of Deep Learning based Token Classification models during Training

TL;DR

This study argues that traditional token-classification metrics such as F1 fail to capture the practical deployability of models in business document processing. It introduces Document Integrity Precision (DIP), a document-level metric that measures the share of documents that can be processed without any manual interventions, and evaluates it alongside F1 on German receipts using a LayoutLMv1-based model. Through two data-splits that simulate different inference environments, the work shows that high F1 does not guarantee automation-friendly outputs, as DIP can be substantially lower when unseen layouts or vendors are present. The findings emphasize the need for business-task oriented metrics to guide model training and deployment, and point to future work on extending DIP to partial correctness and other tasks beyond token classification.

Abstract

This paper shows that further evaluation metrics during model training are needed to decide about its applicability in inference. As an example, a LayoutLM-based model is trained for token classification in documents. The documents are German receipts. We show that conventional classification metrics, represented by the F1-Score in our experiments, are insufficient for evaluating the applicability of machine learning models in practice. To address this problem, we introduce a novel metric, Document Integrity Precision (DIP), as a solution for visual document understanding and the token classification task. To the best of our knowledge, nothing comparable has been introduced in this context. DIP is a rigorous metric, describing how many documents of the test dataset require manual interventions. It enables AI researchers and software developers to conduct an in-depth investigation of the level of process automation in business software. In order to validate DIP, we conduct experiments with our created models to highlight and analyze the impact and relevance of DIP to evaluate if the model should be deployed or not in different training settings. Our results demonstrate that existing metrics barely change for isolated model impairments, whereas DIP indicates that the model requires substantial human interventions in deployment. The larger the set of entities being predicted, the less sensitive conventional metrics are, entailing poor automation quality. DIP, in contrast, remains a single value to be interpreted for entire entity sets. This highlights the importance of having metrics that focus on the business task for model training in production. Since DIP is created for the token classification task, more research is needed to find suitable metrics for other training tasks.

Figures (5)

• Figure 1: Visualized exemplary output of a token classification model. The boxes indicate tokens with labels of interest such as the name of the creditor (red), the invoice number (blue) or the invoice date (green). These tokens could be used for data-analysis or further processing, like automatically filling out archive-systems or web-forms.
• Figure 2: Overview of TCT system for a document $d_i$ with $n$ tokens and $c$ possible token classes
• Figure 3: Distribution fileamount of creditors. For up to 10 files, the histogram has a range per bin of 1, from 10-50 files of 10 and afterwards 50.
• Figure 4: Comparison of the avg. F1-Scores and DIP for each scenario.
• Figure 5: Failure cases: Extracts from sample invoices of run S1_100 whose predictions for label creditorname, grossamount and invoicenumber are respectively wrong. DIP indicates that 20% of all documents need to be manually corrected. However, in terms of individual labels, this effect is averaged out, resulting in overall high F1 scores, which leads to a falsely very good assessment of the prediction quality.