OccamLLM: Fast and Exact Language Model Arithmetic in a Single Step

TL;DR

OccamLLM addresses the persistent arithmetic limitations of LLMs by fusing a frozen LLM with an interpretable symbolic network, OccamNet, enabling exact arithmetic in a single autoregressive step without finetuning or executing generated code. A per-token OccamNet initializer and a learned switch route arithmetic between the language model and the symbolic calculator, yielding fast, secure, and interpretable computations. Empirical results show $100\%$ accuracy on single arithmetic operations and strong performance on broad mathematical problem solving benchmarks, with 70B variants often outperforming GPT-4o and GPT-4o + Code Interpreter while using far fewer generation tokens. This approach demonstrates a scalable path for empowering smaller LLMs with precise arithmetic and suggests broader potential for tight, interpretable tool integration in large-scale language models.

Abstract

Despite significant advancements in text generation and reasoning, Large Language Models (LLMs) still face challenges in accurately performing complex arithmetic operations. Language model systems often enable LLMs to generate code for arithmetic operations to achieve accurate calculations. However, this approach compromises speed and security, and fine-tuning risks the language model losing prior capabilities. We propose a framework that enables exact arithmetic in a single autoregressive step, providing faster, more secure, and more interpretable LLM systems with arithmetic capabilities. We use the hidden states of a LLM to control a symbolic architecture that performs arithmetic. Our implementation using Llama 3 with OccamNet as a symbolic model (OccamLlama) achieves 100\% accuracy on single arithmetic operations ($+,-,\times,÷,\sin{},\cos{},\log{},\exp{},\sqrt{}$), outperforming GPT 4o with and without a code interpreter. Furthermore, OccamLlama outperforms GPT 4o with and without a code interpreter on average across a range of mathematical problem solving benchmarks, demonstrating that OccamLLMs can excel in arithmetic tasks, even surpassing much larger models. We will make our code public shortly.

Figures (7)

• Figure 1: The OccamLLM system. For each autoregressive step, the language model hidden states for that token are fed into a decoder block which assigns weights to OccamNet. The system feeds the most recent numbers from the text into OccamNet, which then evaluates the sparse function specified by its weights. The decoder then determines whether to use the LLM output or the OccamNet output.
• Figure 2: a) A schematic of the OccamNet architecture, with softmax layers in grey and their outputs in red. b) A Directed Acyclic Graph (DAG) (with edges not connected to the output removed for clarity) formed by sampling from OccamNet. This DAG corresponds to the function $\sin(\sin(x_1)\cdot \exp(x_0)).$ Modified from OccamNet.
• Figure 3: Accuracy of OccamLlama and baselines on mathematical problem solving tasks. Higher is better. OccamLlama 8B achieves accuracy comparable to Llama 3 8B on benchmarks with simple arithmetic, higher accuracy than GPT 4o and GPT 4o + Code on on tasks with challenging arithmetic, and accuracy above Llama 3 8B and similar to GPT 3.5 Turbo on average. OccamLlama 70B outperforms GPT 4o and GPT 4o + Code on average.
• Figure 4: Examples from Llama 3 8B Instruct and OccamLlama 8B on (top) an arithmetic problem and (bottom) a mathematical reasoning problem from the MultiArith Float dataset. In OccamLlama, the LLM performs reasoning, the switch predicts when to use OccamNet, and OccamNet performs arithmetic operations. OccamNet's inputs and outputs are highlighted in purple and green, respectively.
• Figure 5: Model performance degradation for each language relative to English in the MGSM dataset. OccamLlama 8B's performance degradation is considerably less than Llama 3 8B's performance degradation, demonstrating strong multilingual and generalization capabilities.