MatheMagic: Generating Dynamic Mathematics Benchmarks Robust to Memorization

TL;DR

MatheMagic addresses the challenge of memorization in mathematical reasoning benchmarks by introducing a dynamic, test-time counterfactual framework in which arithmetic rules are procedurally transformed across $9$ dimensions. The approach uses seed-controlled generation to produce labeled, verifiable problems that test both inductions (inference of the rule from examples) and deductions (application of an explicit rule). Across multiple models, the findings show a clear gap: models excel when an explicit rule is provided but struggle to infer rules from examples, and fine-tuning does not yield robust inductive generalization, often leading to memorization of patterns rather than transferable reasoning. The work suggests a practical path toward contamination-resistant evaluation and highlights the need for methods that promote genuine inductive reasoning beyond memorization in mathematical tasks.

Abstract

Conducting contamination-free evaluation of mathematical capabilities can be difficult for two reasons: models may memorize a test set once it is made public, and current mathematical benchmarks are prone to overfitting due to having limited diversity of symbols and rules, coupled with closed-ended answers. This paper proposes a method to leverage these shortcomings as useful features to a construct dynamic, counterfactual benchmark, which can be used to both reveal overfitting and measure true reasoning. We demonstrate this via MatheMagic, which generates math test instances with the interpretations of numbers and operators altered, yet has automatically verifiable answers. Test instances are randomly seeded and constructed at test time to evaluate a model's induction or deduction capability, offering stability, extensibility, comparability, and robustness to overfitting. Our experiments find that models solve deduction more easily than induction, but they revert to standard math. Further analysis reveals that math-adapted models fail to exhibit a general "skill" of reasoning, and fine-tuning on induction tasks generalizes poorly.

Figures (9)

• Figure 1: Performance of in-context learning (inductive reasoning) across shots.
• Figure 2: Model performance in inductive versus deductive reasoning at 512 shots on MatheMagic.
• Figure 3: Qwen 72B results by transformation type at 512 shots, in both inductive and deductive settings.
• Figure 4: Breakdown of response types for Qwen 72B in the inductive setting at various shots. The bars show the percentage of responses that are: a correct answer, a reversion error (reverting to the original math), a novel error (incorrect under both new and original rules), or a format error.
• Figure 5: Percentage of answers that are correct w.r.t. original math, reported across shots.