Plutus: Benchmarking Large Language Models in Low-Resource Greek Finance

TL;DR

This work tackles the scarcity of Greek financial NLP resources by introducing Plutus-ben, the first Greek financial evaluation benchmark, and Plutus-8B, a Greek financial LLM fine-tuned on domain data. Plutus-ben defines five core Greek financial tasks (numeric NER, textual NER, QA, abstractive summarization, and topic classification) and provides three new Greek datasets (GRFinNUM, GRFinNER, GRFinQA) along with two existing resources (GRFNS-2023, GRMultiFin); it also presents instruction-tuning data and a comprehensive evaluation protocol. A broad evaluation of 22 LLMs reveals that Greek financial tasks are challenging due to linguistic complexity and domain-specific terminology, with cross-lingual transfer offering limited benefits. Fine-tuning on Greek financial data yields clear gains, culminating in Plutus-8B achieving SOTA performance on Plutus-ben, particularly in numeric reasoning and entity recognition, though long-form summarization remains a bottleneck. The authors publicize all resources to promote reproducible research and advance Greek financial NLP, underscoring the importance of language- and domain-specific adaptation to achieve robust performance in non-English finance tasks.

Abstract

Despite Greece's pivotal role in the global economy, large language models (LLMs) remain underexplored for Greek financial context due to the linguistic complexity of Greek and the scarcity of domain-specific datasets. Previous efforts in multilingual financial natural language processing (NLP) have exposed considerable performance disparities, yet no dedicated Greek financial benchmarks or Greek-specific financial LLMs have been developed until now. To bridge this gap, we introduce Plutus-ben, the first Greek Financial Evaluation Benchmark, and Plutus-8B, the pioneering Greek Financial LLM, fine-tuned with Greek domain-specific data. Plutus-ben addresses five core financial NLP tasks in Greek: numeric and textual named entity recognition, question answering, abstractive summarization, and topic classification, thereby facilitating systematic and reproducible LLM assessments. To underpin these tasks, we present three novel, high-quality Greek financial datasets, thoroughly annotated by expert native Greek speakers, augmented by two existing resources. Our comprehensive evaluation of 22 LLMs on Plutus-ben reveals that Greek financial NLP remains challenging due to linguistic complexity, domain-specific terminology, and financial reasoning gaps. These findings underscore the limitations of cross-lingual transfer, the necessity for financial expertise in Greek-trained models, and the challenges of adapting financial LLMs to Greek text. We release Plutus-ben, Plutus-8B, and all associated datasets publicly to promote reproducible research and advance Greek financial NLP, fostering broader multilingual inclusivity in finance.

Figures (6)

• Figure 1: Radar graph of model performance on Plutus-ben, the first Greek financial benchmark. Plutus-8B achieves the best performance, surpassing GPT-4 by 15.38%, GPT-4o by 46.34%, and Deepseek-V3 by 93.55%.
• Figure 2: Comparison of model win rates in fluency, coherence, and factuality between Plutus-8B and GPT-4, evaluated on the GRFNS-2023 dataset within the Plutus-ben benchmark.
• Figure 3: The Plutus-ben interface.
• Figure 4: The Demo of Plutus-8B-instruct.
• Figure 5: The Label Studio interface of the NER annotation process.