Evaluating Semantic and Syntactic Understanding in Large Language Models for Payroll Systems

TL;DR

The paper addresses the reliability of general-purpose LLMs in payroll-style tasks that fuse semantic understanding with exact numeric computation. It introduces a task-focused evaluation using a synthetic payroll dataset, tiered rule graphs, and prompt-level controls across multiple model families, comparing prompt-only to tool-augmented approaches. Results show that simple arithmetic and short rule graphs are handled well with minimal prompting, but multi-jurisdiction, currency conversion, and policy caps demand explicit, auditable computation paths—often via generated code or external tools. The proposed reproducible framework provides practical guidance for deploying LLMs in regulated, high-stakes settings and clarifies the regimes where computation augmentation is necessary for auditability.

Abstract

Large language models are now used daily for writing, search, and analysis, and their natural language understanding continues to improve. However, they remain unreliable on exact numerical calculation and on producing outputs that are straightforward to audit. We study synthetic payroll system as a focused, high-stakes example and evaluate whether models can understand a payroll schema, apply rules in the right order, and deliver cent-accurate results. Our experiments span a tiered dataset from basic to complex cases, a spectrum of prompts from minimal baselines to schema-guided and reasoning variants, and multiple model families including GPT, Claude, Perplexity, Grok and Gemini. Results indicate clear regimes where careful prompting is sufficient and regimes where explicit computation is required. The work offers a compact, reproducible framework and practical guidance for deploying LLMs in settings that demand both accuracy and assurance.