Exploring the Vertical-Domain Reasoning Capabilities of Large Language Models

TL;DR

This paper defines vertical-domain accounting reasoning and argues that domain-specific reasoning is the core capability required for LLMs in accounting. It introduces an accounting-focused evaluation framework with three reasoning dimensions (mathematical reasoning, accounting knowledge reasoning, and comprehensive accounting reasoning) and constructs Chinese-domain benchmarks together with a Multi-Calculation Benchmark derived from MR-GSM8K to test multi-step numeric and rule-based inferences. It evaluates GPT-4 and GLM-series (e.g., GLM-6B, GLM-130B, GLM-4) under Few-shot-CoT prompting, finding that Few-shot-CoT yields about $50\%$ gains in accuracy, with GPT-4 achieving the strongest general reasoning and domain-specific accounting accuracy remaining modest (e.g., $16.58\%$ for GPT-4 and $21.78\%$ for GLM-4 on specialized tasks). The results reveal a gap to real-world deployment and point to future directions in domain-adaptive data, reasoning-aware model design, and integrated domain knowledge for reliable enterprise accounting systems.

Abstract

Large Language Models (LLMs) are reshaping learning paradigms, cognitive processes, and research methodologies across a wide range of domains. Integrating LLMs with professional fields and redefining the relationship between LLMs and domain-specific applications has become a critical challenge for promoting enterprise digital transformation and broader social development. To effectively integrate LLMs into the accounting domain, it is essential to understand their domain-specific reasoning capabilities. This study introduces the concept of vertical-domain accounting reasoning and establishes evaluation criteria by analyzing the training data characteristics of representative GLM-series models. These criteria provide a foundation for subsequent research on reasoning paradigms and offer benchmarks for improving accounting reasoning performance. Based on this framework, we evaluate several representative models, including GLM-6B, GLM-130B, GLM-4, and OpenAI GPT-4, on a set of accounting reasoning tasks. Experimental results show that different prompt engineering strategies lead to varying degrees of performance improvement across models, with GPT-4 achieving the strongest accounting reasoning capability. However, current LLMs still fall short of real-world application requirements. In particular, further optimization is needed for deployment in enterprise-level accounting scenarios to fully realize the potential value of LLMs in this domain.

Figures (3)

• Figure 1: Multi-Calculation Benchmark: 3-shot Chain-of-Thought Evaluation Results.
• Figure 2: Accounting-Reasoning-Benchmark (3-shot CoT) Evaluation Results.
• Figure 3: Comparison of Error Type Proportions between Normal and Medium Difficulty Levels.