Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring the Meta-level Reasoning of Large Language Models via a Tool-based Multi-hop Tabular Question Answering Task

Nick Ferguson, Alan Bundy, Kwabena Nuamah

TL;DR

The paper investigates meta-level reasoning in LLMs by distinguishing meta- from object-level processes and introducing a tool-based multi-hop tabular QA task built on World Bank indicators. It defines essential actions and a tool-calling loop to quantify meta-level reasoning beyond final accuracy, evaluating off-the-shelf models without fine-tuning. Findings suggest that several models exhibit strong meta-level reasoning with correct tool selection, but persistent gaps in task understanding and numeracy emerge when arithmetic tools are limited, underscoring the need for external symbolic arithmetic for robust object-level reasoning. The work provides a generalizable evaluation framework and dataset for analyzing meta-level reasoning in LLMs, with implications for QA systems in data-driven tasks and guidance for broader domain testing and future research directions.

Abstract

Recent advancements in Large Language Models (LLMs) are increasingly focused on "reasoning" ability, a concept with many overlapping definitions in the LLM discourse. We take a more structured approach, distinguishing meta-level reasoning (denoting the process of reasoning about intermediate steps required to solve a task) from object-level reasoning (which concerns the low-level execution of the aforementioned steps.) We design a novel question answering task, which is based around the values of geopolitical indicators for various countries over various years. Questions require breaking down into intermediate steps, retrieval of data, and mathematical operations over that data. The meta-level reasoning ability of LLMs is analysed by examining the selection of appropriate tools for answering questions. To bring greater depth to the analysis of LLMs beyond final answer accuracy, our task contains 'essential actions' against which we can compare the tool call output of LLMs to infer the strength of reasoning ability. We find that LLMs demonstrate good meta-level reasoning on our task, yet are flawed in some aspects of task understanding. We find that n-shot prompting has little effect on accuracy; error messages encountered do not often deteriorate performance; and provide additional evidence for the poor numeracy of LLMs. Finally, we discuss the generalisation and limitation of our findings to other task domains.

Exploring the Meta-level Reasoning of Large Language Models via a Tool-based Multi-hop Tabular Question Answering Task

TL;DR

The paper investigates meta-level reasoning in LLMs by distinguishing meta- from object-level processes and introducing a tool-based multi-hop tabular QA task built on World Bank indicators. It defines essential actions and a tool-calling loop to quantify meta-level reasoning beyond final accuracy, evaluating off-the-shelf models without fine-tuning. Findings suggest that several models exhibit strong meta-level reasoning with correct tool selection, but persistent gaps in task understanding and numeracy emerge when arithmetic tools are limited, underscoring the need for external symbolic arithmetic for robust object-level reasoning. The work provides a generalizable evaluation framework and dataset for analyzing meta-level reasoning in LLMs, with implications for QA systems in data-driven tasks and guidance for broader domain testing and future research directions.

Abstract

Recent advancements in Large Language Models (LLMs) are increasingly focused on "reasoning" ability, a concept with many overlapping definitions in the LLM discourse. We take a more structured approach, distinguishing meta-level reasoning (denoting the process of reasoning about intermediate steps required to solve a task) from object-level reasoning (which concerns the low-level execution of the aforementioned steps.) We design a novel question answering task, which is based around the values of geopolitical indicators for various countries over various years. Questions require breaking down into intermediate steps, retrieval of data, and mathematical operations over that data. The meta-level reasoning ability of LLMs is analysed by examining the selection of appropriate tools for answering questions. To bring greater depth to the analysis of LLMs beyond final answer accuracy, our task contains 'essential actions' against which we can compare the tool call output of LLMs to infer the strength of reasoning ability. We find that LLMs demonstrate good meta-level reasoning on our task, yet are flawed in some aspects of task understanding. We find that n-shot prompting has little effect on accuracy; error messages encountered do not often deteriorate performance; and provide additional evidence for the poor numeracy of LLMs. Finally, we discuss the generalisation and limitation of our findings to other task domains.
Paper Structure (30 sections, 2 figures, 3 tables, 1 algorithm)

This paper contains 30 sections, 2 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Reasoning in LLMs
  4. Tool-use
  5. Existing Datasets
  6. Meta- and Object-Level Reasoning
  7. Our Dataset
  8. Question Generation
  9. Sourcing Data
  10. Generation of Solutions and Essential Actions
  11. Unanswerable Questions
  12. Experimental Setup
  13. Tool Creation
  14. Tool Calling Loop
  15. Results
  16. ...and 15 more sections

Figures (2)

  • Figure 1: Overview of our question generation and evaluation process. 1 We instantiate question templates with slot values. 2 Using a hand-created templated sequence of required steps and the set of tools, we compute essential actions and answers. 3 Instantiated questions are passed to an LLM, which is held in a loop making tool calls which are executed and returned to the model. 4 The predicted set of tool calls are compared to the essential actions.
  • Figure 2: Comparison of experimental results: (a) effect of error presence on final answer accuracy, and (b) zero-shot accuracy with all tools vs. data retrieval only.