A Tale of LLMs and Induced Small Proxies: Scalable Agents for Knowledge Mining

TL;DR

Knowledge mining at scale faces prohibitive LLM costs and brittle traditional pipelines. Falconer replaces end-to-end LLM execution with a planner–generator–metamodel stack that uses two primitive operations, get_label and get_span, to decompose instructions and train compact proxies. The framework achieves near-LLM instruction-following accuracy while cutting inference cost by up to 90 percent and accelerating large-scale processing by more than 20x. This combination enables scalable, instruction-driven knowledge mining across large corpora and diverse domains.

Abstract

At the core of Deep Research is knowledge mining, the task of extracting structured information from massive unstructured text in response to user instructions. Large language models (LLMs) excel at interpreting such instructions but are prohibitively expensive to deploy at scale, while traditional pipelines of classifiers and extractors remain efficient yet brittle and unable to generalize to new tasks. We introduce Falconer, a collaborative framework that combines the agentic reasoning of LLMs with lightweight proxy models for scalable knowledge mining. In Falconer, LLMs act as planners, decomposing user instructions into executable pipelines, and as annotators, generating supervision to train small proxies. The framework unifies classification and extraction into two atomic operations, get label and get span, enabling a single instruction-following model to replace multiple task-specific components. To evaluate the consistency between proxy models incubated by Falconer and annotations provided by humans and large models, we construct new benchmarks covering both planning and end-to-end execution. Experiments show that Falconer closely matches state-of-the-art LLMs in instruction-following accuracy while reducing inference cost by up to 90% and accelerating large-scale knowledge mining by more than 20x, offering an efficient and scalable foundation for Deep Research.

Figures (6)

• Figure 1: Falconer decomposes the instruction Extract all laptop prices from positive Amazon reviews into get_label and get_spans, generates supervision for training the competent proxy, and executes these primitives efficiently with small-model inference. On the right, we show how Falconer instantiates the subtasks: first classifying reviews as positive laptop reviews, then extracting the corresponding price spans. This design enables Falconer to combine the instruction-following ability of LLMs with the efficiency of small models.
• Figure 2: Classification Pretraining
• Figure 3: Results on NER Datasets with Ground Truth labels
• Figure 4: Performance on labeled dataset of Single Task w/ new metamodel and Consecutive Task w/ metamodel from previous task. Performance on unlabeled dataset of Single Task w/ new metamodel and Consecutive Task w/ metamodel from previous task
• Figure 5: Performance of different number of metamodel for different task type. Performance of different pretraining strategy