An Adaptive Framework for Generating Systematic Explanatory Answer in Online Q&A Platforms

TL;DR

Empirical evaluations underscore SynthRAG's effectiveness, demonstrating its superiority in handling complex questions, overcoming the limitations of naive RAG models, and significantly improving answer quality and depth.

Abstract

Question Answering (QA) systems face challenges in handling complex questions that require multi-domain knowledge synthesis. The naive RAG models, although effective in information retrieval, struggle with complex questions that require comprehensive and in-depth answers. The pioneering task is defined as explanatory answer generation, which entails handling identified challenges such as the requirement for comprehensive information and logical coherence within the generated context. To address these issues, we refer to systematic thinking theory and propose SynthRAG, an innovative framework designed to enhance QA performance. SynthRAG improves on conventional models by employing adaptive outlines for dynamic content structuring, generating systematic information to ensure detailed coverage, and producing customized answers tailored to specific user inquiries. This structured approach guarantees logical coherence and thorough integration of information, yielding responses that are both insightful and methodically organized. Empirical evaluations underscore SynthRAG's effectiveness, demonstrating its superiority in handling complex questions, overcoming the limitations of naive RAG models, and significantly improving answer quality and depth. Furthermore, an online deployment on the Zhihu platform revealed that SynthRAG's answers achieved notable user engagement, with each response averaging 5.73 upvotes and surpassing the performance of 79.8% of human contributors, highlighting the practical relevance and impact of the proposed framework. Our code is available at https://github.com/czy1999/SynthRAG .

Figures (6)

• Figure 1: An illustration of naive RAG and SynthRAG. SynthRAG adeptly integrates and organizes vast, fragmented information, creating a comprehensive and systematic structure.
• Figure 2: The SynthRAG framework consists of three primary components: adaptive outline generation, systematic chapter-level information generation, and customized answer generation. The integration of these components within the SynthRAG framework addresses the fragmentation and incompleteness issues of traditional RAG methods, providing a robust solution for explanatory answer generation tasks.
• Figure 3: Results of reward model evaluation. The reward model-based ordinal evaluation demonstrates the superiority of SynthRAG over baseline models and ablation version. SynthRAG consistently achieves higher user preference scores, indicating its effectiveness in generating answers that align more closely with user tastes.
• Figure 4: Comparison of Like Distributions for Human Answers and SynthRAG Generated Answers on Zhihu.
• Figure 5: Various metrics such as upvotes, engagement, and follower growth of our zhihu bot account.