Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning to Reduce: Optimal Representations of Structured Data in Prompting Large Language Models

Younghun Lee, Sungchul Kim, Tong Yu, Ryan A. Rossi, Xiang Chen

TL;DR

This work tackles the difficulty of prompting LLMs with extensive structured data by introducing Learning to Reduce, a framework that trains a lightweight language model as a policy to generate a reduced, evidence-focused input context for a fixed LLM. The policy is learned with On-Policy Reinforcement Learning (PPO), using annotated table QA data to identify relevant rows and columns, and is applied to both column and row reductions. Empirical results show competitive context-reduction recall and strong generalization to unseen data, with notable improvements on downstream QA tasks when the context is long. The approach offers a cost-efficient, generalizable pre-prompting tool for structured data QA and highlights the potential of integrating LLM inferences as reward signals in future work.

Abstract

Large Language Models (LLMs) have been widely used as general-purpose AI agents showing comparable performance on many downstream tasks. However, existing work shows that it is challenging for LLMs to integrate structured data (e.g. KG, tables, DBs) into their prompts; LLMs need to either understand long text data or select the most relevant evidence prior to inference, and both approaches are not trivial. In this paper, we propose a framework, Learning to Reduce, that fine-tunes a language model to generate a reduced version of an input context, given a task description and context input. The model learns to reduce the input context using On-Policy Reinforcement Learning and aims to improve the reasoning performance of a fixed LLM. Experimental results illustrate that our model not only achieves comparable accuracies in selecting the relevant evidence from an input context, but also shows generalizability on different datasets. We further show that our model helps improve the LLM's performance on downstream tasks especially when the context is long.

Learning to Reduce: Optimal Representations of Structured Data in Prompting Large Language Models

TL;DR

This work tackles the difficulty of prompting LLMs with extensive structured data by introducing Learning to Reduce, a framework that trains a lightweight language model as a policy to generate a reduced, evidence-focused input context for a fixed LLM. The policy is learned with On-Policy Reinforcement Learning (PPO), using annotated table QA data to identify relevant rows and columns, and is applied to both column and row reductions. Empirical results show competitive context-reduction recall and strong generalization to unseen data, with notable improvements on downstream QA tasks when the context is long. The approach offers a cost-efficient, generalizable pre-prompting tool for structured data QA and highlights the potential of integrating LLM inferences as reward signals in future work.

Abstract

Large Language Models (LLMs) have been widely used as general-purpose AI agents showing comparable performance on many downstream tasks. However, existing work shows that it is challenging for LLMs to integrate structured data (e.g. KG, tables, DBs) into their prompts; LLMs need to either understand long text data or select the most relevant evidence prior to inference, and both approaches are not trivial. In this paper, we propose a framework, Learning to Reduce, that fine-tunes a language model to generate a reduced version of an input context, given a task description and context input. The model learns to reduce the input context using On-Policy Reinforcement Learning and aims to improve the reasoning performance of a fixed LLM. Experimental results illustrate that our model not only achieves comparable accuracies in selecting the relevant evidence from an input context, but also shows generalizability on different datasets. We further show that our model helps improve the LLM's performance on downstream tasks especially when the context is long.
Paper Structure (14 sections, 5 equations, 2 figures, 2 tables)

This paper contains 14 sections, 5 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Approach
  3. Reducing Input Context
  4. Training Pipeline
  5. LM as a Policy Network
  6. RL Agent Training
  7. Model Architecture
  8. Models and Experiments
  9. Baseline Models
  10. Performance and Generalizability
  11. Downstream Task Performance
  12. Related Work
  13. Discussion and Conclusion
  14. Experiment Details

Figures (2)

  • Figure 1: Comparison between original prompting (dotted arrow path) and Learning to Reduce (solid arrow path). Given a pair of input question and context, a language model (blue hexagon) learns a policy to generate the relevant rows and columns by getting rewards. The model gets a high negative reward $c_n$ for the number of necessary items that are missing from the prediction. By updating the policy network parameters with rewards, our model generates the correct reduced context which leads the fixed LLM model (e.g. GPT-4) to perform more accurately on downstream tasks.
  • Figure 2: Accuracy of GPT-4 model on WTQ test set with different input context tables. Reducing both rows and columns (red, purple) is more powerful when the context is longer.