Enhancing Quantitative Reasoning Skills of Large Language Models through Dimension Perception
Yuncheng Huang, Qianyu He, Jiaqing Liang, Sihang Jiang, Yanghua Xiao, Yunwen Chen
TL;DR
This work addresses the deficit in LLM quantitative reasoning due to missing dimensional knowledge by introducing a dimension-aware framework built around DimUnitKB, a unit-linking system (DimKS), and the DimEval benchmark. It formalizes dimensional concepts, develops a comprehensive, bilingual dimensional unit knowledge base, and links textual mentions to dimensional units, enabling dimension-aware reasoning. By finetuning with DimEval and augmenting data for Q-MWP, the approach substantially improves performance on quantitative reasoning tasks (e.g., from 43.55% to 50.67% on Q-Ape210k vs GPT-4) and demonstrates robust gains even for smaller models. The results underscore the practical impact of embedding dimensional understanding into LLMs, with potential applications across mathematics, science, engineering, and finance where units and dimensions are central.
Abstract
Quantities are distinct and critical components of texts that characterize the magnitude properties of entities, providing a precise perspective for the understanding of natural language, especially for reasoning tasks. In recent years, there has been a flurry of research on reasoning tasks based on large language models (LLMs), most of which solely focus on numerical values, neglecting the dimensional concept of quantities with units despite its importance. We argue that the concept of dimension is essential for precisely understanding quantities and of great significance for LLMs to perform quantitative reasoning. However, the lack of dimension knowledge and quantity-related benchmarks has resulted in low performance of LLMs. Hence, we present a framework to enhance the quantitative reasoning ability of language models based on dimension perception. We first construct a dimensional unit knowledge base (DimUnitKB) to address the knowledge gap in this area. We propose a benchmark DimEval consisting of seven tasks of three categories to probe and enhance the dimension perception skills of LLMs. To evaluate the effectiveness of our methods, we propose a quantitative reasoning task and conduct experiments. The experimental results show that our dimension perception method dramatically improves accuracy (43.55%->50.67%) on quantitative reasoning tasks compared to GPT-4.