ANLS* -- A Universal Document Processing Metric for Generative Large Language Models
David Peer, Philemon Schöpf, Volckmar Nebendahl, Alexander Rietzler, Sebastian Stabinger
TL;DR
This work introduces ANLS*, a universal evaluation metric for generative large language models applied to document processing, extending ANLS to handle dictionaries, lists, and complex structures while penalizing unanswerable or hallucinated outputs. It defines ANLS* as $ANLS^*(g,p) = \frac{s(g,p)}{l(g,p)}$ with a recursively computed score $s$ and a normalization length $l$, using Levenshtein distance and Hungarian matching, and a fixed threshold $\tau = 0.5$. The authors demonstrate ANLS* across seven datasets and 20+ GLLMs using QA and information-extraction prompts, showing that the SFT prompting method often yields the best results and that vision-based prompts lag behind OCR-enhanced approaches. They also compare against DocLLM and open-source models, finding that large, text-based models with document-tailored prompting generally outperform specialized smaller models, arguing for broader adoption of ANLS* as a common evaluation standard in document processing with GLLMs.
Abstract
Traditionally, discriminative models have been the predominant choice for tasks like document classification and information extraction. These models make predictions that fall into a limited number of predefined classes, facilitating a binary true or false evaluation and enabling the direct calculation of metrics such as the F1 score. However, recent advancements in generative large language models (GLLMs) have prompted a shift in the field due to their enhanced zero-shot capabilities, which eliminate the need for a downstream dataset and computationally expensive fine-tuning. However, evaluating GLLMs presents a challenge as the binary true or false evaluation used for discriminative models is not applicable to the predictions made by GLLMs. This paper introduces a new metric for generative models called ANLS* for evaluating a wide variety of tasks, including information extraction and classification tasks. The ANLS* metric extends existing ANLS metrics as a drop-in-replacement and is still compatible with previously reported ANLS scores. An evaluation of 7 different datasets, and more than 20 different GLLMs together with 3 different prompting methods using the ANLS* metric is also provided, demonstrating the importance of the proposed metric. We also benchmark a novel approach to generate prompts for documents, called SFT, against other prompting techniques such as LATIN. In almost all cases, SFT outperforms other techniques and improves the state-of-the-art, sometimes by as much as $10$ percentage points. Sources are available at https://github.com/deepopinion/anls_star_metric