Fine-Tuning Language Models to Know What They Know

TL;DR

This work introduces ESMA, a gradient-free fine-tuning framework that binds LLM internal knowledge to explicit behavior via a joint Direct/Meta-question reward. By measuring metacognition with $d'_{\rm type2}$ and related metrics, the authors show ESMA yields robust generalization across unseen data, languages, and novel information, with improvements driven by a sparse subset of weight updates. The findings demonstrate that metacognitive alignment is achievable with evolution strategies and highlight practical implications for producing more reliable, knowledge-aware AI systems. The study also provides evidence against simple reward hacking and suggests directions for more efficient, cross-domain metacognition research.

Abstract

Metacognition is a critical component of intelligence, specifically regarding the awareness of one's own knowledge. While humans rely on shared internal memory for both answering questions and reporting their knowledge state, this dependency in LLMs remains underexplored. This study proposes a framework to measure metacognitive ability $d_{\rm{type2}}'$ using a dual-prompt method, followed by the introduction of Evolution Strategy for Metacognitive Alignment (ESMA) to bind a model's internal knowledge to its explicit behaviors. ESMA demonstrates robust generalization across diverse untrained settings, indicating a enhancement in the model's ability to reference its own knowledge. Furthermore, parameter analysis attributes these improvements to a sparse set of significant modifications.

Figures (18)

• Figure 1: Overview of Evolution Strategy for Metacognitive Alignment (ESMA). The process begins with an initial parent LLM, whose weights are perturbed with Gaussian noise to create a population of model variants. Each variant is evaluated on a dual-axis task: a Direct Question (to test factual knowledge) and a Meta Question (to test self-knowledge). A Joint Reward is calculated based on the alignment between the correctness and meta responses, measuring whether the model knows what it knows. Finally, the next-generation LLM is produced by a weighted average of the variants, prioritizing higher rewards, and the cycle repeats.
• Figure 2: Type 2 ROC Curves across model scales. The plots compare the Original (orange) and ESMA (blue) models using continuous confidence scores. ESMA consistently improves metacognitive sensitivity across all scales, shifting performance toward the high metacognition regime (AUC $\approx$ 0.75). This demonstrates that the alignment gained through ESMA remains robust on a continuous scale even when trained on binary outputs.
• Figure 3: Distribution of metacognitive confidence by correctness across model scales. Density of confidence scores $D$ are plotted for the original and ESMA of the 1.5B, 3B, and 7B Qwen2.5 models. Blue and orange denote correct and incorrect meta-answers, respectively. ESMA results in a marked shift in all models: the overlap between distributions decreases, and the confidence scores shift toward 1.0 for the correct answers and 0.0 for the incorrect answers. In this manner, the improved metacognitive abilities observed in \ref{['tab:overall-result']} are based on fundamental changes in the LLM's internal representations and processes.
• Figure 4: Effect of weight patching ratio on metacognitive abilities. The plot illustrates how $d_{\rm{type2}}'$ (blue bars, right axis) and raw alignment (%, orange bars, left axis) change as the top $p\%$ of weight updates (by L1 magnitude) are applied to the Qwen2.5 1.5B model. Significant gains were observed with only the top 10% of updates, after which performance plateaued. This observation suggests that metacognitive improvements are driven by a sparse subset of parameters.
• Figure 5: Effect of weight patching ratio on metacognitive abilities