Robust Uncertainty Quantification for Self-Evolving Large Language Models via Continual Domain Pretraining

TL;DR

This work tackles reliable uncertainty quantification for self-evolving LLMs under continual domain pretraining (CDP). It introduces Adaptive Rejection and Non-Exchangeable CP (AR-NECP), a model-agnostic framework that both abstains from answering when competence shifts and aligns calibration with unknown test-domain distributions using transformer-based domain clustering. By reweighting or resampling calibration data and employing label-conditional rejection, AR-NECP achieves $1-\alpha$ coverage while producing compact prediction sets, even under domain shift and catastrophic forgetting. Empirical evaluations on TriviaQA, HotPotQA, and MMLU with multiple LLMs demonstrate improved coverage reliability and prediction efficiency, along with theoretical insights into weighted and resampling CP under non-exchangeable CDP conditions.

Abstract

Continual Learning (CL) is essential for enabling self-evolving large language models (LLMs) to adapt and remain effective amid rapid knowledge growth. Yet, despite its importance, little attention has been given to establishing statistical reliability guarantees for LLMs under CL, particularly in the setting of continual domain pretraining (CDP). Conformal Prediction (CP) has shown promise in offering correctness guarantees for LLMs, but it faces major challenges in CDP: testing data often stems from unknown or shifting domain distributions, under which CP may no longer provide valid guarantees. Moreover, when high coverage is required, CP can yield excessively large prediction sets for unanswerable queries, reducing informativeness. To address these challenges, we introduce an adaptive rejection and non-exchangeable CP framework. Our method first estimates the distribution of questions across domains in the test set using transformer-based clustering, then reweights or resamples the calibration data accordingly. Building on this, adaptive rejection CP allows the LLM to selectively abstain from answering when its confidence or competence shifts significantly. Extensive experiments demonstrate that our framework enhances both the effectiveness and reliability of CP under CDP scenarios. Our code is available at: https://anonymous.4open.science/r/CPCL-8C12/

Figures (19)

• Figure 1: Different quantiles across domains (i.e., "sc", "ht", "sp", "en", "gg") in TriviaQA for NE (left), indicating uncertainty about whether the LLM can answer, and the answer uncertainty (right), measuring the likelihood that the LLM will produce the ground-truth answer. $x$-axis is quantile and $y$-axis is quantile value for the uncertainty measurement. "whole" represents average quantile. We see that quantile values vary among different domains. Results are from Llama-3.1-8B-Instruct.
• Figure 2: Coverage of the three CP methods under different quantiles, different distribution shifts, and using different balancing methods (resampling, reweighting, and non-balancing). Results that lie closest to the $1-\alpha$ line indicate that the corresponding methods achieve the desired performance, neither exceeding it (overcoverage) nor falling short (undercoverage).
• Figure 3: Average prediction set size of the three CP methods under different quantiles, under different distribution shifts, and using different balancing methods. A smaller value means better performance.
• Figure 4: Coverage of the three CP methods under different quantiles under different distribution shifts and using different balancing methods. Results for Mistral7B on TriviaQA.
• Figure 5: Average prediction set size of the three CP methods under different quantiles, under different distribution shifts, and using different balancing methods. Results for Mistral7B on TriviaQA.