Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Effective Framework to Help Large Language Models Handle Numeric-involved Long-context Tasks

Yijiong Yu

TL;DR

This paper tackles the challenge that large language models struggle with numeric-involved long-context tasks. It introduces a structured workflow that decomposes tasks into data extraction and analysis, using small models for extraction and LLM-generated code executed externally for arithmetic, thereby reducing token costs while maintaining or improving accuracy. Evaluations on two benchmarks show substantial improvements in accuracy and significant cost savings, demonstrating scalability and practicality for long-context QA with numerical data. The approach offers a flexible, parallelizable framework that mitigates arithmetic limitations of LLMs and lowers API usage, with limitations related to applicability and data-extraction reliability.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in handling long texts and have almost perfect performance in traditional retrieval tasks. However, their performance significantly degrades when it comes to numerical calculations in the long-context. Numeric-involved long-context tasks typically cannot be addressed by current LLMs in normal settings due to their inherent limitations in simultaneously handling complex and massive information. Some CoT like prompting methods can improve accuracy but demands massive output tokens, which is costly and slow. To address this issue, we propose a workflow, which decompose a numeric-involved long-context task into 4 low-level subtasks: judging, extracting and processing with code and conclusion. The former 2 subtasks is relatively simple, which allows us to use smaller models for efficiently processing long context. When numerical calculations are required, we use code generated by LLMs to avoid the disadvantage of LLM not being good at calculations. The results in 2 numeric-involved long-context benchmarks demonstrate our workflow can not only improve accuracy, but also significantly reduce the cost of API calls.

An Effective Framework to Help Large Language Models Handle Numeric-involved Long-context Tasks

TL;DR

This paper tackles the challenge that large language models struggle with numeric-involved long-context tasks. It introduces a structured workflow that decomposes tasks into data extraction and analysis, using small models for extraction and LLM-generated code executed externally for arithmetic, thereby reducing token costs while maintaining or improving accuracy. Evaluations on two benchmarks show substantial improvements in accuracy and significant cost savings, demonstrating scalability and practicality for long-context QA with numerical data. The approach offers a flexible, parallelizable framework that mitigates arithmetic limitations of LLMs and lowers API usage, with limitations related to applicability and data-extraction reliability.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in handling long texts and have almost perfect performance in traditional retrieval tasks. However, their performance significantly degrades when it comes to numerical calculations in the long-context. Numeric-involved long-context tasks typically cannot be addressed by current LLMs in normal settings due to their inherent limitations in simultaneously handling complex and massive information. Some CoT like prompting methods can improve accuracy but demands massive output tokens, which is costly and slow. To address this issue, we propose a workflow, which decompose a numeric-involved long-context task into 4 low-level subtasks: judging, extracting and processing with code and conclusion. The former 2 subtasks is relatively simple, which allows us to use smaller models for efficiently processing long context. When numerical calculations are required, we use code generated by LLMs to avoid the disadvantage of LLM not being good at calculations. The results in 2 numeric-involved long-context benchmarks demonstrate our workflow can not only improve accuracy, but also significantly reduce the cost of API calls.

Paper Structure

This paper contains 16 sections, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Method
  3. Analyze the question
  4. Segment
  5. Remove irrelevant parts
  6. Segment again
  7. Extract data
  8. Process data
  9. Conclusion
  10. Experiment
  11. Evaluation Dataset
  12. Model Settings
  13. Results
  14. Conclusion
  15. Limitations
  16. ...and 1 more sections

Figures (1)

  • Figure 1: The structure of our method for numeric-involved long-context tasks.