FinNuE: Exposing the Risks of Using BERTScore for Numerical Semantic Evaluation in Finance
Yu-Shiang Huang, Yun-Yu Lee, Tzu-Hsin Chou, Che Lin, Chuan-Ju Wang
TL;DR
This work demonstrates that BERTScore, a prevailing embedding-based metric, largely ignores numerical magnitude in financial text, risking misleading evaluations in finance where a small numeric change can have large interpretive impact. The authors introduce FinNuE, a diagnostic dataset with controlled numerical perturbations drawn from diverse financial sources, and evaluate BERTScore under anchor-based and cross-pair protocols. Across two checkpoints (bert-base-uncased and FinBERT), results show high triplet accuracy in random settings but substantial drops with rule-based perturbations, a steep decline in Kendall's $\tau_b$ for ranking, and near-random cross-context accuracy, indicating global insensitivity to numbers. The study argues for numerically-aware evaluation frameworks—improving tokenization, encoding numerical relations, or hybrid embedding-numerical comparison—crucially informing finance-oriented NLP research and practice, and commits to releasing FinNuE for community use.
Abstract
BERTScore has become a widely adopted metric for evaluating semantic similarity between natural language sentences. However, we identify a critical limitation: BERTScore exhibits low sensitivity to numerical variation, a significant weakness in finance where numerical precision directly affects meaning (e.g., distinguishing a 2% gain from a 20% loss). We introduce FinNuE, a diagnostic dataset constructed with controlled numerical perturbations across earnings calls, regulatory filings, social media, and news articles. Using FinNuE, demonstrate that BERTScore fails to distinguish semantically critical numerical differences, often assigning high similarity scores to financially divergent text pairs. Our findings reveal fundamental limitations of embedding-based metrics for finance and motivate numerically-aware evaluation frameworks for financial NLP.