Synthetic Data-Driven Prompt Tuning for Financial QA over Tables and Documents

TL;DR

This paper tackles the challenge of numerical financial QA over long tables and documents by introducing a self‑improving, data‑augmented prompt framework. It combines a Fin‑Generator, Fin‑Verifiers, and Fin‑Prompt Optimizer in a closed loop to generate synthetic data, verify its quality, and iteratively refine a prompt without external labels. Empirical results on DocMath‑Eval show robust gains in accuracy and robustness over standard prompting methods, including across long and short contexts and multiple LLMs. The approach offers a scalable path to more reliable financial reasoning with LLMs and reduces dependence on labeled data, enabling adaptation to evolving document formats and question types.

Abstract

Financial documents like earning reports or balance sheets often involve long tables and multi-page reports. Large language models have become a new tool to help numerical reasoning and understanding these documents. However, prompt quality can have a major effect on how well LLMs perform these financial reasoning tasks. Most current methods tune prompts on fixed datasets of financial text or tabular data, which limits their ability to adapt to new question types or document structures, or they involve costly and manually labeled/curated dataset to help build the prompts. We introduce a self-improving prompt framework driven by data-augmented optimization. In this closed-loop process, we generate synthetic financial tables and document excerpts, verify their correctness and robustness, and then update the prompt based on the results. Specifically, our framework combines a synthetic data generator with verifiers and a prompt optimizer, where the generator produces new examples that exposes weaknesses in the current prompt, the verifiers check the validity and robustness of the produced examples, and the optimizer incrementally refines the prompt in response. By iterating these steps in a feedback cycle, our method steadily improves prompt accuracy on financial reasoning tasks without needing external labels. Evaluation on DocMath-Eval benchmark demonstrates that our system achieves higher performance in both accuracy and robustness than standard prompt methods, underscoring the value of incorporating synthetic data generation into prompt learning for financial applications.

Figures (2)

• Figure 1: Overview of our proposed iterative prompt optimization workflow. Starting from the left, the Fin-Generator creates a synthetic financial question-answer pair at a specified difficulty level $c$. The generated data then moves to the Fin-Verifiers, which checks its validity and robustness. If the data fails verification, it returns to the Fin-Generator to regenerate a new sample. Once verified, the data proceeds to the Fin-Prompt Optimizer, which systematically improves the current prompt through three sub-steps: analyzing errors, recommending targeted improvements, and revising accordingly. The newly revised prompt is tested against both current and previously resolved cases. If errors persist, the prompt is refined further; if it passes successfully, the system moves on to generate more challenging synthetic data at a higher difficulty level until no error remains.
• Figure 2: Our approach begins with a base prompt (top-left) and shortened version of its refined prompt synthesized on Long (bottom-left). For comparison, the baseline methods—Chain-of-Thought (top-right) and Program-of-Thought (bottom-right)—are shown on the right side of the figure.