FinReflectKG - EvalBench: Benchmarking Financial KG with Multi-Dimensional Evaluation

TL;DR

FinReflectKG-EvalBench tackles the absence of a standard benchmark for financial knowledge graph extraction from SEC filings by introducing a bias-aware, multi-extraction-mode framework. It combines a deterministic LLM-as-Judge with a commit-then-justify protocol and strict bias controls to evaluate candidate triples across faithfulness $F$, precision $P$, relevance $R$, and comprehensiveness $C$, using micro- and macro-averaging. Empirical results show reflection-based extraction provides the most balanced coverage (highest $C$, $P$, and $R$) while single-pass yields the strongest faithfulness, highlighting a fundamental trade-off among extraction quality dimensions. The framework enables transparent benchmarking, error analysis, and iterative improvement for financial AI applications, with future work extending coverage beyond the S&P 100 and diversifying document types to enhance governance and reliability.

Abstract

Large language models (LLMs) are increasingly being used to extract structured knowledge from unstructured financial text. Although prior studies have explored various extraction methods, there is no universal benchmark or unified evaluation framework for the construction of financial knowledge graphs (KG). We introduce FinReflectKG - EvalBench, a benchmark and evaluation framework for KG extraction from SEC 10-K filings. Building on the agentic and holistic evaluation principles of FinReflectKG - a financial KG linking audited triples to source chunks from S&P 100 filings and supporting single-pass, multi-pass, and reflection-agent-based extraction modes - EvalBench implements a deterministic commit-then-justify judging protocol with explicit bias controls, mitigating position effects, leniency, verbosity and world-knowledge reliance. Each candidate triple is evaluated with binary judgments of faithfulness, precision, and relevance, while comprehensiveness is assessed on a three-level ordinal scale (good, partial, bad) at the chunk level. Our findings suggest that, when equipped with explicit bias controls, LLM-as-Judge protocols provide a reliable and cost-efficient alternative to human annotation, while also enabling structured error analysis. Reflection-based extraction emerges as the superior approach, achieving best performance in comprehensiveness, precision, and relevance, while single-pass extraction maintains the highest faithfulness. By aggregating these complementary dimensions, FinReflectKG - EvalBench enables fine-grained benchmarking and bias-aware evaluation, advancing transparency and governance in financial AI applications.

Figures (1)

• Figure 1: System prompt design