Patchfinder: Leveraging Visual Language Models for Accurate Information Retrieval using Model Uncertainty

TL;DR

The paper addresses extracting precise fields (latitude, longitude, TVD) from noisy scanned documents without relying on fine-tuned OCR+LLM pipelines.It proposes PatchFinder, a patch-based approach that uses Patch Confidence derived from Maximum Softmax Probability to guide patch size and obtain confidence-weighted predictions from a Phi-3.5-Vision–based Vision-Language Model.Key results show PatchFinder achieving $94\%$ accuracy on 190 historical well documents, outperforming ChatGPT-4o by $18.5$ percentage points, and extending to noisy financial statements and standard benchmarks like CORD and FUNSD.This approach demonstrates that uncertainty-driven patching can enable end-to-end information extraction from degraded documents on widely accessible hardware.

Abstract

For decades, corporations and governments have relied on scanned documents to record vast amounts of information. However, extracting this information is a slow and tedious process due to the sheer volume and complexity of these records. The rise of Vision Language Models (VLMs) presents a way to efficiently and accurately extract the information out of these documents. The current automated workflow often requires a two-step approach involving the extraction of information using optical character recognition software and subsequent usage of large language models for processing this information. Unfortunately, these methods encounter significant challenges when dealing with noisy scanned documents, often requiring computationally expensive language models to handle high information density effectively. In this study, we propose PatchFinder, an algorithm that builds upon VLMs to improve information extraction. First, we devise a confidence-based score, called Patch Confidence, based on the Maximum Softmax Probability of the VLMs' output to measure the model's confidence in its predictions. Using this metric, PatchFinder determines a suitable patch size, partitions the input document into overlapping patches, and generates confidence-based predictions for the target information. Our experimental results show that PatchFinder, leveraging Phi-3v, a 4.2-billion-parameter VLM, achieves an accuracy of 94% on our dataset of 190 noisy scanned documents, outperforming ChatGPT-4o by 18.5 percentage points.

Figures (5)

• Figure 1: Illustration of PatchFinder's Confidence-Based prediction. The input image is first partitioned into multiple patches ($m=1,2,3$ are shown as examples). Then, visual tokens of each patch are obtained via an image encoder, i.e., CLIP ViT-L/14 radford2021learning. Those tokens are then combined with the prompting-text-tokens in an interleaved way, and fed through a transformer decoder, i.e., Phi-3-mini-128K-instruct abdin2024phi3technicalreporthighly. Using the transformer's MSP outputs, PatchFinder computes the Patch Confidence and selects the most confident patch. The target sequence is then generated with maximum-likelihood criteria on that patch. More details in Subsect. \ref{['subsect:details']}.
• Figure 2: Patch Confidence as a function of patch size in the Colorado and Pennsylvania development split (20 documents) of the scanned well documents dataset.
• Figure 3: Examples of different drilling completion reports and well records. It can be seen that the data is not only highly different in terms of fonts, formats, and structure; their target information is also represented with different indicators.
• Figure 4: Correlation between Accuracy and Patch Confidence along different patch sizes.
• Figure 5: Qualitative Examples. Green indicates a correct token, red indicates an incorrect token.