Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Stage Field Extraction of Financial Documents with OCR and Compact Vision-Language Models

Yichao Jin, Yushuo Wang, Qishuai Zhong, Kent Chiu Jin-Chun, Kenneth Zhu Ke, Donald MacDonald

TL;DR

This work addresses the challenge of extracting structured financial indicators from long, multilingual SMB documents that are often only available as noisy scanned images. It presents a multistage pipeline that combines image pre-processing, multilingual OCR, BM25-based page retrieval, and compact Vision-Language Models to perform targeted field extraction on narrowed-down pages. The approach yields about 8.8x higher field-level accuracy than a one-shot large VLM baseline, while using only 0.7% of the GPU resources and achieving roughly 92.6% lower end-to-end latency, highlighting significant efficiency and scalability gains for in-house financial processing. The framework offers interpretable outputs with traceability to source pages and demonstrates practical impact for institutions handling large volumes of confidential, multilingual documents.

Abstract

Financial documents are essential sources of information for regulators, auditors, and financial institutions, particularly for assessing the wealth and compliance of Small and Medium-sized Businesses. However, SMB documents are often difficult to parse. They are rarely born digital and instead are distributed as scanned images that are none machine readable. The scans themselves are low in resolution, affected by skew or rotation, and often contain noisy backgrounds. These documents also tend to be heterogeneous, mixing narratives, tables, figures, and multilingual content within the same report. Such characteristics pose major challenges for automated information extraction, especially when relying on end to end large Vision Language Models, which are computationally expensive, sensitive to noise, and slow when applied to files with hundreds of pages. We propose a multistage pipeline that leverages traditional image processing models and OCR extraction, together with compact VLMs for structured field extraction of large-scale financial documents. Our approach begins with image pre-processing, including segmentation, orientation detection, and size normalization. Multilingual OCR is then applied to recover page-level text. Upon analyzing the text information, pages are retrieved for coherent sections. Finally, compact VLMs are operated within these narrowed-down scopes to extract structured financial indicators. Our approach is evaluated using an internal corpus of multi-lingual, scanned financial documents. The results demonstrate that compact VLMs, together with a multistage pipeline, achieves 8.8 times higher field level accuracy relative to directly feeding the whole document into large VLMs, only at 0.7 percent of the GPU cost and 92.6 percent less end-to-end service latency.

Multi-Stage Field Extraction of Financial Documents with OCR and Compact Vision-Language Models

TL;DR

This work addresses the challenge of extracting structured financial indicators from long, multilingual SMB documents that are often only available as noisy scanned images. It presents a multistage pipeline that combines image pre-processing, multilingual OCR, BM25-based page retrieval, and compact Vision-Language Models to perform targeted field extraction on narrowed-down pages. The approach yields about 8.8x higher field-level accuracy than a one-shot large VLM baseline, while using only 0.7% of the GPU resources and achieving roughly 92.6% lower end-to-end latency, highlighting significant efficiency and scalability gains for in-house financial processing. The framework offers interpretable outputs with traceability to source pages and demonstrates practical impact for institutions handling large volumes of confidential, multilingual documents.

Abstract

Financial documents are essential sources of information for regulators, auditors, and financial institutions, particularly for assessing the wealth and compliance of Small and Medium-sized Businesses. However, SMB documents are often difficult to parse. They are rarely born digital and instead are distributed as scanned images that are none machine readable. The scans themselves are low in resolution, affected by skew or rotation, and often contain noisy backgrounds. These documents also tend to be heterogeneous, mixing narratives, tables, figures, and multilingual content within the same report. Such characteristics pose major challenges for automated information extraction, especially when relying on end to end large Vision Language Models, which are computationally expensive, sensitive to noise, and slow when applied to files with hundreds of pages. We propose a multistage pipeline that leverages traditional image processing models and OCR extraction, together with compact VLMs for structured field extraction of large-scale financial documents. Our approach begins with image pre-processing, including segmentation, orientation detection, and size normalization. Multilingual OCR is then applied to recover page-level text. Upon analyzing the text information, pages are retrieved for coherent sections. Finally, compact VLMs are operated within these narrowed-down scopes to extract structured financial indicators. Our approach is evaluated using an internal corpus of multi-lingual, scanned financial documents. The results demonstrate that compact VLMs, together with a multistage pipeline, achieves 8.8 times higher field level accuracy relative to directly feeding the whole document into large VLMs, only at 0.7 percent of the GPU cost and 92.6 percent less end-to-end service latency.
Paper Structure (22 sections, 5 figures, 1 table)

This paper contains 22 sections, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. OCR and Preprocessing
  4. Text and Layout Understanding
  5. Financial Document Analysis
  6. Vision-Language Models
  7. Problem Statement and Methodology
  8. Problem Statement
  9. Multi-stage Parsing Pipeline
  10. Image Pre-processing
  11. Multi-lingual OCR Transcription
  12. Page Retrieval
  13. Extraction using Compact VLM
  14. Experiments
  15. Dataset
  16. ...and 7 more sections

Figures (5)

  • Figure 1: Overview of the proposed multi-stage parsing framework. It begins with page-level splitting and image pre-processing, followed by multilingual OCR transcription. A BM25-based retrieval step filters relevant pages into semantic categories such as company background, financial tables, etc. Depending on the category, either an LLM is applied for information summarization or a compact VLM is used for pre-defined entity extraction. The outputs are merged into structured data.
  • Figure 2: The illustration of image pre-processing task. It consists of three steps, including page segmentation, deskew and rotation correction, and image re-normailzation. These steps are critical to the performance of the downstream OCR and information extraction tasks.
  • Figure 3: Sample OCR transcription output showing recognized text, confidence scores, and bounding box coordinates for layout-aware extraction.
  • Figure 4: Characteristics of the in-house experimental dataset with 93 scanned financial documents from SMBs. (i) Left: document lengths distribution (before and after page retrieval). (ii) Right: document language distribution.
  • Figure 5: Field-level accuracy calculation example based on five pre-defined fields.