Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Equipping Language Models with Tool Use Capability for Tabular Data Analysis in Finance

Adrian Theuma, Ehsan Shareghi

TL;DR

This work demonstrates the feasibility of equipping a mid-sized language model with tool use for finance-focused tabular data analysis by fine-tuning only a tiny fraction of parameters. The Raven framework uses a Task Router and Task Solver to dynamically decide whether to answer in‑branch or to invoke offline tools (a calculator and a lightweight SQL engine), enabling accurate, multi-hop reasoning over structured data. Empirical results across four finance QA benchmarks show substantial gains, including a 63.8 percentage-point improvement on Wiki-SQL and an average 35.2% uplift over the base model, with Raven outperforming GPT-3.5 on average. The study highlights the practical viability of tool-augmented LMs for finance, while noting hardware limitations and the potential for scale with larger models and longer context lengths.

Abstract

Large language models (LLMs) have exhibited an array of reasoning capabilities but face challenges like error propagation and hallucination, particularly in specialised areas like finance, where data is heterogeneous, and precision is paramount. We explore the potential of language model augmentation with external tools to mitigate these limitations and offload certain reasoning steps to external tools that are more suited for the task, instead of solely depending on the LLM's inherent abilities. More concretely, using financial domain question-answering datasets, we apply supervised fine-tuning on a LLaMA-2 13B Chat model to act both as a 'task router' and 'task solver'. The 'task router' dynamically directs a question to either be answered internally by the LLM or externally via the right tool from the tool set. Our tool-equipped SFT model, Raven, demonstrates an improvement of 35.2% and 5.06% over the base model and SFT-only baselines, respectively, and is highly competitive with strong GPT-3.5 results. To the best of our knowledge, our work is the first that investigates tool augmentation of language models for the finance domain.

Equipping Language Models with Tool Use Capability for Tabular Data Analysis in Finance

TL;DR

This work demonstrates the feasibility of equipping a mid-sized language model with tool use for finance-focused tabular data analysis by fine-tuning only a tiny fraction of parameters. The Raven framework uses a Task Router and Task Solver to dynamically decide whether to answer in‑branch or to invoke offline tools (a calculator and a lightweight SQL engine), enabling accurate, multi-hop reasoning over structured data. Empirical results across four finance QA benchmarks show substantial gains, including a 63.8 percentage-point improvement on Wiki-SQL and an average 35.2% uplift over the base model, with Raven outperforming GPT-3.5 on average. The study highlights the practical viability of tool-augmented LMs for finance, while noting hardware limitations and the potential for scale with larger models and longer context lengths.

Abstract

Large language models (LLMs) have exhibited an array of reasoning capabilities but face challenges like error propagation and hallucination, particularly in specialised areas like finance, where data is heterogeneous, and precision is paramount. We explore the potential of language model augmentation with external tools to mitigate these limitations and offload certain reasoning steps to external tools that are more suited for the task, instead of solely depending on the LLM's inherent abilities. More concretely, using financial domain question-answering datasets, we apply supervised fine-tuning on a LLaMA-2 13B Chat model to act both as a 'task router' and 'task solver'. The 'task router' dynamically directs a question to either be answered internally by the LLM or externally via the right tool from the tool set. Our tool-equipped SFT model, Raven, demonstrates an improvement of 35.2% and 5.06% over the base model and SFT-only baselines, respectively, and is highly competitive with strong GPT-3.5 results. To the best of our knowledge, our work is the first that investigates tool augmentation of language models for the finance domain.
Paper Structure (18 sections, 3 figures, 1 table)

This paper contains 18 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Raven
  3. Fine-tuning Data
  4. Tools
  5. Instruction Tuning
  6. Inference
  7. Experiments
  8. Main Results
  9. Analysis
  10. Conclusion
  11. Background on LMs in Finance
  12. Training Details
  13. Templates
  14. TAT-QA
  15. PhraseBank
  16. ...and 3 more sections

Figures (3)

  • Figure 1: Raven Inference Flow. Using the language model the Task Router infers the optimal task format to use conditioned on the given prompt. The Task Solver re-formats the instruction according to the selected template by the task router and sends it to the language model again. The pipeline will branch between serving the response directly or calling a tool API to perform an intermediate evaluation before serving the final output.
  • Figure 2: Comparison of model performance on the TAT-QA dataset specifically highlighting the effect of a tools-augmented model on questions that require multi-hop reasoning.
  • Figure 3: Comparison of model performance on the TAT-QA dataset highlighting the effect of complexity on model performance.