DIMSUM: Discourse in Mathematical Reasoning as a Supervision Module
Krish Sharma, Niyar R Barman, Akshay Chaturvedi, Nicholas Asher
TL;DR
DIMSUM investigates whether current LLM progress on GSM8K reflects genuine mathematical reasoning or memorization. It introduces discourse-structure as a supervision module and demonstrates that encoding semantic relations among premises yields substantial improvements, including large gains on out-of-distribution variants. The authors construct Hard GSM8K and three transformation variants (C-MOD, N-MOD, L-MOD) to probe robustness, showing that discourse signals benefit both small and large models and mitigate memorization effects. The work highlights the practical value of discourse-aware supervision for robust mathematical reasoning and discusses limitations and future directions, such as annotating discourse structure for smaller models and avoiding irrelevant content in evaluation data.
Abstract
We look at reasoning on GSM8k, a dataset of short texts presenting primary school, math problems. We find, with Mirzadeh et al. (2024), that current LLM progress on the data set may not be explained by better reasoning but by exposure to a broader pretraining data distribution. We then introduce a novel information source for helping models with less data or inferior training reason better: discourse structure. We show that discourse structure improves performance for models like Llama2 13b by up to 160%. Even for models that have most likely memorized the data set, adding discourse structural information to the model still improves predictions and dramatically improves large model performance on out of distribution examples.