Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MTFinEval:A Multi-domain Chinese Financial Benchmark with Eurypalynous questions

Xinyu Liu, Ke Jin

TL;DR

MTFinEval addresses the need for evaluating theoretical economic knowledge in financial LLMs beyond task-specific benchmarks. It constructs a 360-question benchmark spanning six economics domains using university textbooks and exam questions, with expert validation to ensure foundational relevance. The authors formulate zero-shot objective functions for true/false, single-choice, and multiple-choice questions and evaluate a set of open-source Chinese-financial LLMs, reporting widespread underperformance as a sign of gaps in economic reasoning. This work provides a robust evaluation framework that informs model selection and guides future improvements to improve domain reliability in rapidly changing economic environments.

Abstract

With the emergence of more and more economy-specific LLMS, how to measure whether they can be safely invested in production becomes a problem. Previous research has primarily focused on evaluating the performance of LLMs within specific application scenarios. However, these benchmarks cannot reflect the theoretical level and generalization ability, and the backward datasets are increasingly unsuitable for problems in real scenarios. In this paper, we have compiled a new benchmark, MTFinEval, focusing on the LLMs' basic knowledge of economics, which can always be used as a basis for judgment. To examine only theoretical knowledge as much as possible, MTFinEval is build with foundational questions from university textbooks,and exam papers in economics and management major. Aware of the overall performance of LLMs do not depend solely on one subdiscipline of economics, MTFinEval comprise 360 questions refined from six major disciplines of economics, and reflect capabilities more comprehensively. Experiment result shows all LLMs perform poorly on MTFinEval, which proves that our benchmark built on basic knowledge is very successful. Our research not only offers guidance for selecting the appropriate LLM for specific use cases, but also put forward increase the rigor reliability of LLMs from the basics.

MTFinEval:A Multi-domain Chinese Financial Benchmark with Eurypalynous questions

TL;DR

MTFinEval addresses the need for evaluating theoretical economic knowledge in financial LLMs beyond task-specific benchmarks. It constructs a 360-question benchmark spanning six economics domains using university textbooks and exam questions, with expert validation to ensure foundational relevance. The authors formulate zero-shot objective functions for true/false, single-choice, and multiple-choice questions and evaluate a set of open-source Chinese-financial LLMs, reporting widespread underperformance as a sign of gaps in economic reasoning. This work provides a robust evaluation framework that informs model selection and guides future improvements to improve domain reliability in rapidly changing economic environments.

Abstract

With the emergence of more and more economy-specific LLMS, how to measure whether they can be safely invested in production becomes a problem. Previous research has primarily focused on evaluating the performance of LLMs within specific application scenarios. However, these benchmarks cannot reflect the theoretical level and generalization ability, and the backward datasets are increasingly unsuitable for problems in real scenarios. In this paper, we have compiled a new benchmark, MTFinEval, focusing on the LLMs' basic knowledge of economics, which can always be used as a basis for judgment. To examine only theoretical knowledge as much as possible, MTFinEval is build with foundational questions from university textbooks,and exam papers in economics and management major. Aware of the overall performance of LLMs do not depend solely on one subdiscipline of economics, MTFinEval comprise 360 questions refined from six major disciplines of economics, and reflect capabilities more comprehensively. Experiment result shows all LLMs perform poorly on MTFinEval, which proves that our benchmark built on basic knowledge is very successful. Our research not only offers guidance for selecting the appropriate LLM for specific use cases, but also put forward increase the rigor reliability of LLMs from the basics.
Paper Structure (9 sections, 3 equations, 3 figures, 1 table)

This paper contains 9 sections, 3 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. MTFinEval Benchmark
  4. Dataset Statistics
  5. Formulation
  6. Experiments
  7. Experimental Setup
  8. Results
  9. Conclusion

Figures (3)

  • Figure 1: Answer pairs on different question types.
  • Figure 2: types and numbers of questions
  • Figure 3: subject and type radars