Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Accounting Reasoning Capabilities of Large Language Models

Jie Zhou, Xin Chen, Jie Zhang, Hai Li, Jie Wang, Zhe Li

TL;DR

The paper defines vertical domain accounting reasoning and proposes a dedicated benchmarking framework to evaluate how large language models perform domain-specific numerical and rule-based tasks. By evaluating GPT-4 and GLM variants on mathematical, knowledge-based, and integrated accounting reasoning tasks—including the Multi-Calculation Benchmark and Chinese CPA-derived data—the study shows that prompt design, particularly Few-shot-CoT, significantly boosts performance, yet current models still struggle with multi-step accounting reasoning and error propagation. Key findings reveal that even the strongest model (GPT-4) falls short of enterprise readiness, especially as reasoning chains lengthen, indicating the need for domain-aligned data, structured reasoning architectures, and targeted training. The work highlights actionable directions for improving practical accounting AI, such as domain adaptation, robust rule application, and improved interpretability of accounting reasoning processes.

Abstract

Large language models are transforming learning, cognition, and research across many fields. Effectively integrating them into professional domains, such as accounting, is a key challenge for enterprise digital transformation. To address this, we define vertical domain accounting reasoning and propose evaluation criteria derived from an analysis of the training data characteristics of representative GLM models. These criteria support systematic study of accounting reasoning and provide benchmarks for performance improvement. Using this framework, we evaluate GLM-6B, GLM-130B, GLM-4, and OpenAI GPT-4 on accounting reasoning tasks. Results show that prompt design significantly affects performance, with GPT-4 demonstrating the strongest capability. Despite these gains, current models remain insufficient for real-world enterprise accounting, indicating the need for further optimization to unlock their full practical value.

Evaluating Accounting Reasoning Capabilities of Large Language Models

TL;DR

The paper defines vertical domain accounting reasoning and proposes a dedicated benchmarking framework to evaluate how large language models perform domain-specific numerical and rule-based tasks. By evaluating GPT-4 and GLM variants on mathematical, knowledge-based, and integrated accounting reasoning tasks—including the Multi-Calculation Benchmark and Chinese CPA-derived data—the study shows that prompt design, particularly Few-shot-CoT, significantly boosts performance, yet current models still struggle with multi-step accounting reasoning and error propagation. Key findings reveal that even the strongest model (GPT-4) falls short of enterprise readiness, especially as reasoning chains lengthen, indicating the need for domain-aligned data, structured reasoning architectures, and targeted training. The work highlights actionable directions for improving practical accounting AI, such as domain adaptation, robust rule application, and improved interpretability of accounting reasoning processes.

Abstract

Large language models are transforming learning, cognition, and research across many fields. Effectively integrating them into professional domains, such as accounting, is a key challenge for enterprise digital transformation. To address this, we define vertical domain accounting reasoning and propose evaluation criteria derived from an analysis of the training data characteristics of representative GLM models. These criteria support systematic study of accounting reasoning and provide benchmarks for performance improvement. Using this framework, we evaluate GLM-6B, GLM-130B, GLM-4, and OpenAI GPT-4 on accounting reasoning tasks. Results show that prompt design significantly affects performance, with GPT-4 demonstrating the strongest capability. Despite these gains, current models remain insufficient for real-world enterprise accounting, indicating the need for further optimization to unlock their full practical value.
Paper Structure (12 sections, 3 figures, 1 table)

This paper contains 12 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Development of LLMs in the Accounting Domain
  4. Reasoning Capability of LLMs in Accounting
  5. Evaluation Benchmarks for Chinese Accounting Reasoning
  6. Experiments
  7. Experimental Setup
  8. Evaluation of Mathematical Reasoning Ability
  9. Evaluation of Chinese Accounting Reasoning Ability
  10. Comprehensive Accounting Reasoning Evaluation
  11. Evaluation Results and Analysis
  12. Conclusion

Figures (3)

  • Figure 1: Multi-Calculation Benchmark: 3-shot Chain-of-Thought Evaluation Results.
  • Figure 2: Accounting-Reasoning-Benchmark (3-shot CoT) Evaluation Results.
  • Figure 3: Comparison of Error Type Proportions between Normal and Medium Difficulty Levels.