MetaRuleGPT: Recursive Numerical Reasoning of Language Models Trained with Simple Rules

TL;DR

MetaRuleGPT tackles the challenge of numerical reasoning in LLMs by learning and composing simple arithmetic and logical rules through a Transformer-based architecture. It uses a rule-based, iterative approach with components such as VeriGate and RefeedFormatter to breakdown problems and ensure accuracy. The paper demonstrates, on arithmetic and vector cross-product tasks, that rule learning can outperform traditional CoT-based reasoning and large baselines, achieving robust 100% accuracy on challenging high-digit problems with a relatively small parameter count. This work suggests that structured rule-learning can significantly enhance numerical reasoning and generalization in language models, with potential for broader mathematical tasks.

Abstract

Recent studies have highlighted the limitations of large language models in mathematical reasoning, particularly their inability to capture the underlying logic. Inspired by meta-learning, we propose that models should acquire not only task-specific knowledge but also transferable problem-solving skills. We introduce MetaRuleGPT, a novel Transformer-based architecture that performs precise numerical calculations and complex logical operations by learning and combining different rules. In contrast with traditional training sets, which are heavily composed of massive raw instance data, MetaRuleGPT is pre-trained on much less abstract datasets containing basic, compound, and iterative rules for mathematical reasoning. Extensive experimental results demonstrate MetaRuleGPT can mimic human's rule-following capabilities, break down complexity, and iteratively derive accurate results for complex mathematical problems. These findings prove the potential of rule learning to enhance the numerical reasoning abilities of language models.

Figures (6)

• Figure 1: This image illustrates the differences between MetaRuleGPT and the traditional Chain-of-Thought (CoT) reasoning method in handling mathematical problems. MetaRuleGPT employs a rule-based reasoning approach, breaking down and solving problems step-by-step through predefined rules, such as alignment rules, carry rules, and borrow rules. This structured method ensures accuracy and generalization ability in reasoning, allowing the model to systematically handle various computational tasks while avoiding the common hallucination errors found in CoT methods.
• Figure 2: This picture shows some training rule examples, where * and ! represents 10 and 9 in decimal respectively. Once our training data covers the logic required for the operation, the LLMs can follow this underlying logic to complete the arithmetic operation.
• Figure 3: This figure shows Architecture Diagram of MetaRuleGPT. MetaRuleGPT Pre-trained model is a model that has learned basic rules. VeriGate is used to identify whether the current decoding meets the expected transformation of the recent decodings. If it does not meet the expectations, it will enter RefeedFormatter to realign and adjust the structure and then use the basic rules of the model again. According to the operation rules, after a limited number of calls to the basic rule model, the final output is obtained.
• Figure 4: MetaRuleGPT Pre-trained Model. SAN represents Self-Attention Network.
• Figure 5: MetaRuleGPT language model calculation example diagram