Self-DC: When to Reason and When to Act? Self Divide-and-Conquer for Compositional Unknown Questions

TL;DR

This work tackles compositional unknown question answering by introducing Self-DC, a confidence-guided divide-and-conquer framework that dynamically chooses between internal reasoning and external retrieval. It introduces CuQA, a dataset of compositional unknown questions, to benchmark the balance between reasoning and acting. Empirical results show Self-DC achieves comparable or better accuracy with significantly fewer external calls than strong retrieval-based baselines, highlighting the value of internal confidence signals. The study demonstrates robust efficiency gains and provides insights into how to calibrate uncertainty and decomposition in LLM-driven QA systems.

Abstract

Previous research has typically concentrated on leveraging the internal knowledge of Large Language Models (LLMs) to answer known questions (i.e., \textit{internal reasoning such as generate-then-read}). In contrast, for questions that fall outside their known scope, these models rely on external knowledge retrieval to provide accurate responses (i.e., \textit{external acting such as retrieve-then-read}). However, few previous works consider the \textit{compositional questions}, which consist of several known and unknown sub-questions, necessitating the dynamic combination of previous two methods (i.e., \textit{internal reasoning and external acting}) to achieve a better trade-off between effectiveness and efficiency. To this end, we introduce a \textbf{Self} \textbf{D}ivide-and-\textbf{C}onquer (\textit{\texttt{Self-DC}}) framework, accompanying with the first \textbf{C}ompositional \textbf{u}nknown \textbf{Q}uestion-\textbf{A}nswering dataset (CuQA). This framework enables LLMs to adaptively choose between using internal knowledge and retrieving external knowledge as needed, resulting in a better trade-off between effectiveness and efficiency. Experimental results on two datasets demonstrate that \textit{\texttt{Self-DC}} can achieve comparable or even better performance with much fewer external calls compared with several strong baselines.

Figures (10)

• Figure 1: A example of compositional questions, in which a unknown question consists of some sub-questions can be answered using known knowledge while other sub-questions necessitate unknown knowledge according to the cutoff date of LLMs.
• Figure 2: Overview of Self-DC: a) retrieve-then-read for unknown questions, b) decompose-and-combination for uncertain questions; and c) generate-then-read for known questions.
• Figure 3: The simplified python implementation details of Self-DC, consisting of several functions: 1) decompose; 2) combine-sub-qas; 3) generate-then-read; and 4) retrieve-then-read.
• Figure 4: The efficiency analysis of different methods using 4o-mini.
• Figure 5: The performance of different choices of $\alpha$ with $\beta$ = 0.1. Left: The performance of different models with confidence type is prob; and Right: The performance of different confidence types (verb or prob) with the same model 4o-mini.