PRISM: Prompt-Refined In-Context System Modelling for Financial Retrieval

TL;DR

PRISM introduces a training-free framework for financial information retrieval that fuses system prompting, in-context learning, and a lightweight multi-agent system. Through extensive ablations, the authors show that a non-agentic setup with a concise, reasoning-focused prompt and document-level ICL yields the strongest ranking performance on FinAgentBench, achieving an NDCG@5 of 0.71818 on a private validation set. The study also provides a thorough feasibility and reproducibility analysis, highlighting latency, token costs, and statistical reliability, and discusses production-readiness limitations of MAS. The findings demonstrate practical potential for production-scale financial retrieval while outlining directions for adaptive segmentation, hybrid retrieval, and more scalable agent architectures to further improve robustness and efficiency.

Abstract

With the rapid progress of large language models (LLMs), financial information retrieval has become a critical industrial application. Extracting task-relevant information from lengthy financial filings is essential for both operational and analytical decision-making. The FinAgentBench dataset formalizes this problem through two tasks: document ranking and chunk ranking. We present PRISM, a training-free framework that integrates refined system prompting, in-context learning (ICL), and a lightweight multi-agent system. Each component is examined extensively to reveal their synergies: prompt engineering provides precise task instructions, ICL supplies semantically relevant few-shot examples, and the multi-agent system models coordinated scoring behaviour. Our best configuration achieves an NDCG@5 of 0.71818 on the restricted validation split. We further demonstrate that PRISM is feasible and robust for production-scale financial retrieval. Its modular, inference-only design makes it practical for real-world use cases. The source code is released at https://bit.ly/prism-ailens.

Figures (6)

• Figure 1: Overview of the proposed PRISM framework.
• Figure 2: Token count distribution of chunks.
• Figure 3: Latency distribution box plots.
• Figure 4: Token distribution bar charts.
• Figure 5: Word count distribution of chunks.