SciAgent: Tool-augmented Language Models for Scientific Reasoning

TL;DR

The paper addresses the challenge of scientific reasoning by shifting from an omniscient solver to a tool-using agent. It introduces MathFunc, a large corpus of math-related functions, and SciAgent, a four-module, tool-augmented model trained to retrieve, understand, and apply domain tools, evaluated on SciToolBench—a multi-domain benchmark with composable toolsets. Key contributions include a cross-retrieval strategy to derive generalized functions, a dense retriever trained on planning signals, and substantial empirical gains over open-source baselines, with tool-use approaching but not yet matching GPT-4 under tool-enabled conditions. The results highlight the practicality and limitations of tool-augmented reasoning, guiding future work on broader domain coverage and toolset robustness across scientific disciplines.

Abstract

Scientific reasoning poses an excessive challenge for even the most advanced Large Language Models (LLMs). To make this task more practical and solvable for LLMs, we introduce a new task setting named tool-augmented scientific reasoning. This setting supplements LLMs with scalable toolsets, and shifts the focus from pursuing an omniscient problem solver to a proficient tool-user. To facilitate the research of such setting, we construct a tool-augmented training corpus named MathFunc which encompasses over 30,000 samples and roughly 6,000 tools. Building on MathFunc, we develop SciAgent to retrieve, understand and, if necessary, use tools for scientific problem solving. Additionally, we craft a benchmark, SciToolBench, spanning five scientific domains to evaluate LLMs' abilities with tool assistance. Extensive experiments on SciToolBench confirm the effectiveness of SciAgent. Notably, SciAgent-Mistral-7B surpasses other LLMs with the same size by more than 13% in absolute accuracy. Furthermore, SciAgent-DeepMath-7B shows much superior performance than ChatGPT.

Figures (12)

• Figure 1: Two paradigms for scientific reasoning. Different colors represent different scientific domains. Left: Collecting annotations and fine-tuning LLMs domain by domain. Right: Our proposed tool-augmented setting. LLMs are fine-tuned on math-related, tool-augmented samples (color in red). When adapting LLMs to a specific domain, a pluggable and domain-specific toolset is attached. No additional fine-tuning is further required.
• Figure 2: Automatic pipeline for MathFunc construction. Please view it starting from the bottom left corner and proceed clockwise. We disentangle the constructions of toolset (dashed lines) and function-augmented samples (solid lines) for more generalized annotations. We do not visualize the function-free samples for simplicity.
• Figure 3: The model architecture of SciAgent. Given a domain-specific toolset , our agent answers the question through four consecutive modules. (1) Planning: provides a high-level plan for this problem. (2) Retrieval: retrieves related functions from attached toolset. (3) Action: generates a low-level solution interleaving rationale and program. The program uses the retrieved functions if necessary. (4) Execution: calls Python executor to run the program and outputs the final answer. Not included in this figure for simplicity.
• Figure 4: Left: Histogram of FPQ (function per question). Higher values indicate greater composability. Right: Histogram of function occurrence. Higher values indicate more generalization and wider application.
• Figure 5: Semi-automatic annotation pipeline for SciToolBench. : GPT-4. : Human annotator.