Collaborative LLM Numerical Reasoning with Local Data Protection

TL;DR

This work presents a privacy-preserving framework for numerical reasoning over documents by combining topic-shifted, pattern-preserving query synthesis with a plug-and-play, tool-based answer reconstruction that reuses remote-model code. By transforming queries to protect sensitive content yet preserving the underlying reasoning skeleton, and by substituting original numerical values after remote solving, the approach leverages remote computation without exposing local data. Empirical results on FinQA and MultiHiertt show substantial improvements in local reasoning accuracy (up to 43.6% gains) while reducing data leakage (up to 44.6%), approaching remote-model performance with strong privacy guarantees. The method generalizes across datasets and local retrievers, offering a practical path for secure, on-device numerical reasoning in real-world applications.

Abstract

Numerical reasoning over documents, which demands both contextual understanding and logical inference, is challenging for low-capacity local models deployed on computation-constrained devices. Although such complex reasoning queries could be routed to powerful remote models like GPT-4, exposing local data raises significant data leakage concerns. Existing mitigation methods generate problem descriptions or examples for remote assistance. However, the inherent complexity of numerical reasoning hinders the local model from generating logically equivalent queries and accurately inferring answers with remote guidance. In this paper, we present a model collaboration framework with two key innovations: (1) a context-aware synthesis strategy that shifts the query topics while preserving reasoning patterns; and (2) a tool-based answer reconstruction approach that reuses the remote-generated plug-and-play solution with code snippets. Experimental results demonstrate that our method achieves better reasoning accuracy than solely using local models while providing stronger data protection than fully relying on remote models. Furthermore, our method improves accuracy by 16.2% - 43.6% while reducing data leakage by 2.3% - 44.6% compared to existing data protection approaches.

Figures (4)

• Figure 1: (a) An example from FinQA with highlighted sensitive information. (b) The typical workflow and challenges for preserving privacy in interactions with remote API-based models.
• Figure 2: The proposed method llustrated with examples for each step. The original query is transformed from one topic (e.g., aircraft fuel consumption) to another (e.g., advertising revenue) by a distilled topic rewriter, while preserving its reasoning pattern (e.g., deriving the total amount from the individual count and percentage). This enables secure delegation to a remote API-based model to elicit a tool-centric, plug-and-play solution for local answer reconstruction through direct numerical switch, without requiring local model re-inference.
• Figure 3: Model cascade with a decision maker. A higher threshold value results in more instances of seeking remote model collaboration. The solid lines represent accuracy, while the dashed lines indicate local data protection.
• Figure 4: Enhancements and deteriorations compared to local self-consistency for the Phi3-mini model on the MultiHiertt dataset. Enhancement refers to cases where local self-consistency was incorrect, but the new method is correct. Deterioration refers to cases where local self-consistency was correct, but the new method is incorrect.