Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring Hybrid Question Answering via Program-based Prompting

Qi Shi, Han Cui, Haofeng Wang, Qingfu Zhu, Wanxiang Che, Ting Liu

TL;DR

Hybrid question answering over heterogeneous data is challenging due to diverse data formats and scales. The paper introduces HProPro, a program-based prompting framework that treats solution discovery as code generation and execution, utilizing function declaration and implementation to enable information-seeking across multiple modalities without domain-specific retrievers or modal transformations. Empirically, HProPro delivers strong few-shot performance on HybridQA and state-of-the-art results on MultiModalQA, with ablations confirming the contributions of the check function, query simplification, and code refinement. This approach offers a transparent, generalizable alternative to modality transformation and retrieval pipelines, with potential broad applicability to real-world, multi-source question answering tasks.

Abstract

Question answering over heterogeneous data requires reasoning over diverse sources of data, which is challenging due to the large scale of information and organic coupling of heterogeneous data. Various approaches have been proposed to address these challenges. One approach involves training specialized retrievers to select relevant information, thereby reducing the input length. Another approach is to transform diverse modalities of data into a single modality, simplifying the task difficulty and enabling more straightforward processing. In this paper, we propose HProPro, a novel program-based prompting framework for the hybrid question answering task. HProPro follows the code generation and execution paradigm. In addition, HProPro integrates various functions to tackle the hybrid reasoning scenario. Specifically, HProPro contains function declaration and function implementation to perform hybrid information-seeking over data from various sources and modalities, which enables reasoning over such data without training specialized retrievers or performing modal transformations. Experimental results on two typical hybrid question answering benchmarks HybridQA and MultiModalQA demonstrate the effectiveness of HProPro: it surpasses all baseline systems and achieves the best performances in the few-shot settings on both datasets.

Exploring Hybrid Question Answering via Program-based Prompting

TL;DR

Hybrid question answering over heterogeneous data is challenging due to diverse data formats and scales. The paper introduces HProPro, a program-based prompting framework that treats solution discovery as code generation and execution, utilizing function declaration and implementation to enable information-seeking across multiple modalities without domain-specific retrievers or modal transformations. Empirically, HProPro delivers strong few-shot performance on HybridQA and state-of-the-art results on MultiModalQA, with ablations confirming the contributions of the check function, query simplification, and code refinement. This approach offers a transparent, generalizable alternative to modality transformation and retrieval pipelines, with potential broad applicability to real-world, multi-source question answering tasks.

Abstract

Question answering over heterogeneous data requires reasoning over diverse sources of data, which is challenging due to the large scale of information and organic coupling of heterogeneous data. Various approaches have been proposed to address these challenges. One approach involves training specialized retrievers to select relevant information, thereby reducing the input length. Another approach is to transform diverse modalities of data into a single modality, simplifying the task difficulty and enabling more straightforward processing. In this paper, we propose HProPro, a novel program-based prompting framework for the hybrid question answering task. HProPro follows the code generation and execution paradigm. In addition, HProPro integrates various functions to tackle the hybrid reasoning scenario. Specifically, HProPro contains function declaration and function implementation to perform hybrid information-seeking over data from various sources and modalities, which enables reasoning over such data without training specialized retrievers or performing modal transformations. Experimental results on two typical hybrid question answering benchmarks HybridQA and MultiModalQA demonstrate the effectiveness of HProPro: it surpasses all baseline systems and achieves the best performances in the few-shot settings on both datasets.
Paper Structure (30 sections, 10 figures, 6 tables)

This paper contains 30 sections, 10 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Method
  3. HProPro Framework
  4. Task Formulation
  5. Function Declaration
  6. Function Implementation
  7. Function Instantiation
  8. Extract information from external source
  9. Compare two pieces of information
  10. Code Refinement
  11. Query Simplification
  12. Experiments
  13. Datasets
  14. Experimental Settings
  15. Baseline Systems
  16. ...and 15 more sections

Figures (10)

  • Figure 1: Example of hybrid question answering task with the corresponding program.
  • Figure 2: Comparison of HProPro with previous retrieval-based methods.
  • Figure 3: Details of the process of the defined functions and the code refinement.
  • Figure 4: Schematic diagram of query simplification process.
  • Figure 5: Error percentage of HProPro on HybridQA and MultiModalQA.
  • ...and 5 more figures