Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Ebisu: Benchmarking Large Language Models in Japanese Finance

Xueqing Peng, Ruoyu Xiang, Fan Zhang, Mingzi Song, Mingyang Jiang, Yan Wang, Lingfei Qian, Taiki Hara, Yuqing Guo, Jimin Huang, Junichi Tsujii, Sophia Ananiadou

TL;DR

Ebisu introduces two expert-annotated tasks, JF-ICR and JF-TE, to assess native Japanese financial language understanding in high-context discourse, focusing on implicit commitment and ground-grounded terminology. Evaluations across 22 LLMs reveal that scaling and generic English-leaning adaptations offer limited gains, while cross-language and domain adaptations do not reliably bridge the gaps in Japanese finance. Annotation quality is high, with robust inter-annotator agreement, and the benchmark exposes persistent challenges in discourse-level intent and nested financial term grounding. By releasing Ebisu’s datasets, guidelines, and evaluation code, the work provides a targeted resource to drive linguistically and culturally informed improvements in financial NLP for Japanese.

Abstract

Japanese finance combines agglutinative, head-final linguistic structure, mixed writing systems, and high-context communication norms that rely on indirect expression and implicit commitment, posing a substantial challenge for LLMs. We introduce Ebisu, a benchmark for native Japanese financial language understanding, comprising two linguistically and culturally grounded, expert-annotated tasks: JF-ICR, which evaluates implicit commitment and refusal recognition in investor-facing Q&A, and JF-TE, which assesses hierarchical extraction and ranking of nested financial terminology from professional disclosures. We evaluate a diverse set of open-source and proprietary LLMs spanning general-purpose, Japanese-adapted, and financial models. Results show that even state-of-the-art systems struggle on both tasks. While increased model scale yields limited improvements, language- and domain-specific adaptation does not reliably improve performance, leaving substantial gaps unresolved. Ebisu provides a focused benchmark for advancing linguistically and culturally grounded financial NLP. All datasets and evaluation scripts are publicly released.

Ebisu: Benchmarking Large Language Models in Japanese Finance

TL;DR

Ebisu introduces two expert-annotated tasks, JF-ICR and JF-TE, to assess native Japanese financial language understanding in high-context discourse, focusing on implicit commitment and ground-grounded terminology. Evaluations across 22 LLMs reveal that scaling and generic English-leaning adaptations offer limited gains, while cross-language and domain adaptations do not reliably bridge the gaps in Japanese finance. Annotation quality is high, with robust inter-annotator agreement, and the benchmark exposes persistent challenges in discourse-level intent and nested financial term grounding. By releasing Ebisu’s datasets, guidelines, and evaluation code, the work provides a targeted resource to drive linguistically and culturally informed improvements in financial NLP for Japanese.

Abstract

Japanese finance combines agglutinative, head-final linguistic structure, mixed writing systems, and high-context communication norms that rely on indirect expression and implicit commitment, posing a substantial challenge for LLMs. We introduce Ebisu, a benchmark for native Japanese financial language understanding, comprising two linguistically and culturally grounded, expert-annotated tasks: JF-ICR, which evaluates implicit commitment and refusal recognition in investor-facing Q&A, and JF-TE, which assesses hierarchical extraction and ranking of nested financial terminology from professional disclosures. We evaluate a diverse set of open-source and proprietary LLMs spanning general-purpose, Japanese-adapted, and financial models. Results show that even state-of-the-art systems struggle on both tasks. While increased model scale yields limited improvements, language- and domain-specific adaptation does not reliably improve performance, leaving substantial gaps unresolved. Ebisu provides a focused benchmark for advancing linguistically and culturally grounded financial NLP. All datasets and evaluation scripts are publicly released.
Paper Structure (53 sections, 18 equations, 4 figures, 4 tables)

This paper contains 53 sections, 18 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Japanese Financial LLMs
  4. Japanese Financial Benchmarks
  5. Ebisu33, 99, 154188, 0, 45 Benchmark
  6. JF-ICR: Japanese Financial Implicit Commitment Recognition
  7. Task Definition.
  8. Data Source.
  9. Expert Annotation.
  10. Quality Validation.
  11. Instruction Data Conversion.
  12. Evaluation Metric.
  13. JF-TE: Japanese Financial Term Extraction
  14. Task Definition
  15. Data Source.
  16. ...and 38 more sections

Figures (4)

  • Figure 1: Representation examples of JF-ICR and JF-TE.
  • Figure 2: Ranked models performance on the Ebisu33, 99, 154188, 0, 45 benchmark..
  • Figure 3: The Label Studio interface of the JF-ICR annotation process.
  • Figure 4: The Label Studio interface of the JF-TE annotation process.