Recursive Question Understanding for Complex Question Answering over Heterogeneous Personal Data

TL;DR

This paper tackles the challenge of answering complex questions over heterogeneous personal data while preserving privacy by running on user devices. It introduces ReQAP, a recursive-question-understanding framework that builds executable operator trees with RETRIEVE and EXTRACT operators to access both structured and unstructured data, enabling traceable answers without global data access. A new benchmark, PerQA, with 20 personas and 3,567+ complex questions across tens of thousands of events per persona, supports rigorous evaluation and realistic data distributions. Empirical results show that ReQAP substantially outperforms verbalization (Rag) and translation-based (CodeGen) baselines, including strong on-device, small-model variants, and user studies confirm the approach aligns with real user information needs and preserves data locality.

Abstract

Question answering over mixed sources, like text and tables, has been advanced by verbalizing all contents and encoding it with a language model. A prominent case of such heterogeneous data is personal information: user devices log vast amounts of data every day, such as calendar entries, workout statistics, shopping records, streaming history, and more. Information needs range from simple look-ups to queries of analytical nature. The challenge is to provide humans with convenient access with small footprint, so that all personal data stays on the user devices. We present ReQAP, a novel method that creates an executable operator tree for a given question, via recursive decomposition. Operators are designed to enable seamless integration of structured and unstructured sources, and the execution of the operator tree yields a traceable answer. We further release the PerQA benchmark, with persona-based data and questions, covering a diverse spectrum of realistic user needs.

Figures (5)

• Figure 1: Excerpt of relevant user data for the question $q^3$, stored in structured (workouts), semi-structured (calendar) and unstructured (social media, mails) sources.
• Figure 2: Example inference of ReQAP. The left side illustrates the output of the QUD stage, an operator tree, for the example question $q^3$. The right side shows how this tree is executed within the OTX stage to derive the answer.
• Figure 3: Example canonicalized event (not accessible at inference), with the corresponding verbalized version.
• Figure 4: LLM instruction for generating operator trees. This is followed by $8$ in-context learning examples, which are dynamically chosen per step. Sample ICL examples are shown in Table \ref{['tab:icl-examples']}.
• Figure 5: LLM instruction for generating complex questions. In addition, we randomly sample $20$ questions that are provided as ICL examples.