Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring the Impact of Table-to-Text Methods on Augmenting LLM-based Question Answering with Domain Hybrid Data

Dehai Min, Nan Hu, Rihui Jin, Nuo Lin, Jiaoyan Chen, Yongrui Chen, Yu Li, Guilin Qi, Yun Li, Nijun Li, Qianren Wang

TL;DR

This work systematically evaluates how four table-to-text generation methods impact LLM-based QA systems operating on domain hybrid data. By constructing ICT-DATA and ICTQA and testing DSFT and RAG frameworks, the study shows that LLM-based and TPLM-based table-to-text yields strongest DSFT performance, while in RAG, LLM-based and Markdown approaches often excel, influenced by term frequency, semantic representations, and retrieval dynamics. The findings reveal that text produced by different methods affects retrieval quality and question answering through both linguistic richness and embedding-space separability. These insights offer practical guidance for deploying domain-specific QA systems with hybrid data, balancing performance, cost, and retrieval considerations.

Abstract

Augmenting Large Language Models (LLMs) for Question Answering (QA) with domain specific data has attracted wide attention. However, domain data often exists in a hybrid format, including text and semi-structured tables, posing challenges for the seamless integration of information. Table-to-Text Generation is a promising solution by facilitating the transformation of hybrid data into a uniformly text-formatted corpus. Although this technique has been widely studied by the NLP community, there is currently no comparative analysis on how corpora generated by different table-to-text methods affect the performance of QA systems. In this paper, we address this research gap in two steps. First, we innovatively integrate table-to-text generation into the framework of enhancing LLM-based QA systems with domain hybrid data. Then, we utilize this framework in real-world industrial data to conduct extensive experiments on two types of QA systems (DSFT and RAG frameworks) with four representative methods: Markdown format, Template serialization, TPLM-based method, and LLM-based method. Based on the experimental results, we draw some empirical findings and explore the underlying reasons behind the success of some methods. We hope the findings of this work will provide a valuable reference for the academic and industrial communities in developing robust QA systems.

Exploring the Impact of Table-to-Text Methods on Augmenting LLM-based Question Answering with Domain Hybrid Data

TL;DR

This work systematically evaluates how four table-to-text generation methods impact LLM-based QA systems operating on domain hybrid data. By constructing ICT-DATA and ICTQA and testing DSFT and RAG frameworks, the study shows that LLM-based and TPLM-based table-to-text yields strongest DSFT performance, while in RAG, LLM-based and Markdown approaches often excel, influenced by term frequency, semantic representations, and retrieval dynamics. The findings reveal that text produced by different methods affects retrieval quality and question answering through both linguistic richness and embedding-space separability. These insights offer practical guidance for deploying domain-specific QA systems with hybrid data, balancing performance, cost, and retrieval considerations.

Abstract

Augmenting Large Language Models (LLMs) for Question Answering (QA) with domain specific data has attracted wide attention. However, domain data often exists in a hybrid format, including text and semi-structured tables, posing challenges for the seamless integration of information. Table-to-Text Generation is a promising solution by facilitating the transformation of hybrid data into a uniformly text-formatted corpus. Although this technique has been widely studied by the NLP community, there is currently no comparative analysis on how corpora generated by different table-to-text methods affect the performance of QA systems. In this paper, we address this research gap in two steps. First, we innovatively integrate table-to-text generation into the framework of enhancing LLM-based QA systems with domain hybrid data. Then, we utilize this framework in real-world industrial data to conduct extensive experiments on two types of QA systems (DSFT and RAG frameworks) with four representative methods: Markdown format, Template serialization, TPLM-based method, and LLM-based method. Based on the experimental results, we draw some empirical findings and explore the underlying reasons behind the success of some methods. We hope the findings of this work will provide a valuable reference for the academic and industrial communities in developing robust QA systems.
Paper Structure (29 sections, 3 equations, 7 figures, 18 tables)

This paper contains 29 sections, 3 equations, 7 figures, 18 tables.

Table of Contents

  1. Introduction
  2. Table-to-Text Generation
  3. Building LLM-based QA Systems with Domain Corpora
  4. Dataset and Evaluation Metrics
  5. Evaluation Dataset
  6. Evaluation Metrics
  7. Experimental Setup
  8. Results
  9. RQ1: How do these methods affect the performance of QA systems?
  10. RQ2: What are the potential reasons for their different performances?
  11. RQ3: Are there practical suggestions for choosing table-to-text methods?
  12. Additional discussion on experimental results
  13. Related Work
  14. Domain Augmented Large Language Models.
  15. Question Answering over Hybrid Data
  16. ...and 14 more sections

Figures (7)

  • Figure 1: Illustration of four domain corpora generation process. Different table-to-text methods are applied to tables of domain documents, generating different text. These generated texts are then merged with the original document texts, yielding different domain corpora.
  • Figure 2: Framework of domain-enhanced QA systems.
  • Figure 3: The scores distribution from human evaluation for the DSFT QA systems based on OPT-6.7B.
  • Figure 4: Comparison of human evaluation scores between QA models using different Table-to-Text methods. 'A vs. B win' indicates the percentage of test set instances where Model A's score surpasses Model B's.
  • Figure 5: A t-SNE visualization of chunk clusters in the embedding space of the RAG system. 'X Chunks' represents chunks related to the query (red star) from the corpus generated by X table-to-text method.
  • ...and 2 more figures