Verbatim Data Transcription Failures in LLM Code Generation: A State-Tracking Stress Test
Mohd Ariful Haque, Kishor Datta Gupta, Mohammad Ashiqur Rahman, Roy George
TL;DR
Verbatim transcription of numeric data into code is critical in security-sensitive workflows but remains brittle for state-of-the-art LLMs. The authors introduce a minimal transcription-to-code benchmark and evaluate exact-string inclusion across multiple models and prompt variants, revealing a sharp degradation in long-horizon copying as the number of constants grows. They identify two failure modes—a capacity-limited transcription ceiling and state-tracking derailment—producing either partial transcripts or zero mentions of the required data, with perfect transcription vanishing beyond $N=300$ in all models examined. The work argues for deterministic tooling, external data loading, and integrity checks to ensure data correctness in AI-assisted coding, and positions the benchmark as a compact stress test for reliability in security-critical software pipelines.
Abstract
Many real-world software tasks require exact transcription of provided data into code, such as cryptographic constants, protocol test vectors, allowlists, and calibration tables. These tasks are operationally sensitive because small omissions or alterations can remain silent while producing syntactically valid programs. This paper introduces a deliberately minimal transcription-to-code benchmark to isolate this reliability concern in LLM-based code generation. Given a list of high-precision decimal constants, a model must generate Python code that embeds the constants verbatim and performs a simple aggregate computation. We describe the prompting variants, evaluation protocol based on exact-string inclusion, and analysis framework used to characterize state-tracking and long-horizon generation failures. The benchmark is intended as a compact stress test that complements existing code-generation evaluations by focusing on data integrity rather than algorithmic reasoning.