MACA: A Framework for Distilling Trustworthy LLMs into Efficient Retrievers

TL;DR

The paper addresses the challenge of answering underspecified enterprise queries with high latency constraints by distilling a metadata-aware LLM re-ranker into a fast, single-pass retriever. It introduces MACA, a two-phase framework that first calibrates a trustworthy LLM teacher using a metadata-aware prompt and then distills its judgments into a compact student via a MACA Judge and MetaFusion objective. The approach uses automatic taxonomy labeling to extract topic, sub-topic, intent, and entities, and trains with a combination of metadata-conditioned ranking (MNRL) and cross-model margin alignment (RCMA). Empirical results on proprietary banking FAQs and BankFAQs show substantial gains over baselines and pretrained encoders, with the distilled students recovering much of the teacher’s performance without per-query LLM calls, indicating strong practical value for regulated domains and retrieval-augmented generation.

Abstract

Modern enterprise retrieval systems must handle short, underspecified queries such as ``foreign transaction fee refund'' and ``recent check status''. In these cases, semantic nuance and metadata matter but per-query large language model (LLM) re-ranking and manual labeling are costly. We present Metadata-Aware Cross-Model Alignment (MACA), which distills a calibrated metadata aware LLM re-ranker into a compact student retriever, avoiding online LLM calls. A metadata-aware prompt verifies the teacher's trustworthiness by checking consistency under permutations and robustness to paraphrases, then supplies listwise scores, hard negatives, and calibrated relevance margins. The student trains with MACA's MetaFusion objective, which combines a metadata conditioned ranking loss with a cross model margin loss so it learns to push the correct answer above semantically similar candidates with mismatched topic, sub-topic, or entity. On a proprietary consumer banking FAQ corpus and BankFAQs, the MACA teacher surpasses a MAFA baseline at Accuracy@1 by five points on the proprietary set and three points on BankFAQs. MACA students substantially outperform pretrained encoders; e.g., on the proprietary corpus MiniLM Accuracy@1 improves from 0.23 to 0.48, while keeping inference free of LLM calls and supporting retrieval-augmented generation.

Figures (4)

• Figure 1: MACA reduces entity drift in short banking queries. The baseline returns the same debit-card FAQ for both queries; MACA selects the correct debit/credit FAQ at rank 1 (no LLM at inference).
• Figure 2: MACA pipeline. Phase I: candidates from the query-to-question view $q{\to}Q$ and the query-to-answer view $q{\to}A$ are fused by RRF and re-ranked by a calibrated MACA Teacher using metadata from the automatic taxonomy creation step (Algorithm \ref{['alg:taxonomy']}). Phase II: the MACA Judge selects a teacher positive $d_T^{+}$ and a student hard negative $d_S^{-}$, and emits a margin $\Delta_T$. These triplets are then used to train a compact student with MNRL and RCMA. At inference time, only the student is used.
• Figure 3: Trustworthiness (Consistency@1, Robustness@1) of four prompts on both datasets; MACA is the most balanced.
• Figure 4: Compact prompt cards for MACA. Left: metadata-aware re-ranker. Right: deterministic judge for triplet selection and margin calibration.