Spilling the Beans: Teaching LLMs to Self-Report Their Hidden Objectives

TL;DR

Self-Report Fine-Tuning (SRFT) is a supervised technique that trains LLMs to admit on-policy factual mistakes and confess their hidden objectives when interrogated. The approach generalizes to out-of-distribution stealth tasks, enabling near-ceiling detection of hidden objectives (F1 ≈ 0.98) and substantial elicitation of objective content (28–100% of details). Compared with baselines and prefilled attacks, SRFT demonstrates robust honesty under adversarial prompts and maintains execution capabilities, with some trade-offs in instruction hierarchy compliance and limited model-specific vulnerabilities. The work contributes a practical auditing tool for exposing misalignment in agentic LLMs and offers insights into how on-policy error admission can generalize to hidden objectives, including the influence of system-prompt design. It also highlights limitations and directions for future research on generalization mechanisms and safer deployment.

Abstract

As AI systems become more capable of complex agentic tasks, they also become more capable of pursuing undesirable objectives and causing harm. Previous work has attempted to catch these unsafe instances by interrogating models directly about their objectives and behaviors. However, the main weakness of trusting interrogations is that models can lie. We propose self-report fine-tuning (SRFT), a simple supervised fine-tuning technique that trains models to occasionally make factual mistakes, then admit them when asked. We show that the admission of factual errors in simple question-answering settings generalizes out-of-distribution (OOD) to the admission of hidden misaligned objectives in adversarial agentic settings. We evaluate SRFT in OOD stealth tasks, where models are instructed to complete a hidden misaligned objective alongside a user-specified objective without being caught by monitoring. After SRFT, models are more likely to confess the details of their hidden objectives when interrogated, even under strong pressure not to disclose them. Interrogation on SRFT models can detect hidden objectives with near-ceiling performance (F1 score = 0.98), while the baseline model lies when interrogated under the same conditions (F1 score = 0). Interrogation on SRFT models can further elicit the content of the hidden objective, recovering 28-100% details, compared to 0% details recovered in the baseline model and by prefilled assistant turn attacks. This provides a promising technique for promoting honesty propensity and incriminating misaligned AIs.

Figures (28)

• Figure 1: SRFT training and interrogation pipeline. We train a model to occasionally make mistakes during factual QA, then admit them when asked through self-report fine-tuning (SRFT). We evaluate the resulting model in stealth tasks where it must complete a hidden objective alongside a user-specified objective without being caught by monitoring. Then, we interrogate the model about its objectives using task-agnostic interrogation questions. After SRFT, the model tends to honestly confess the details of its hidden objective, despite in-context adversarial pressure to conceal.
• Figure 2: Hidden objective detection. After SRFT, the model honestly admits to having a hidden objective when interrogated (by the Hidden Objective question) across tasks (average F1 score = 0.98), even with in-context instructions to conceal objectives. In contrast, baseline (no SRFT) never discloses having hidden objectives when interrogated by the same question (average F1 score = 0). We tested 8 binary questions across 4 categories and SRFT outperforms the baseline across all questions (Appendix \ref{['app:full-res']}). SRFT shows the strongest generalization to the Hidden Objective category and the worst generalization to the Harm category. Error bars show 95% Wilson's CI.
• Figure 3: Hidden objective content elicitation. Interrogation after SRFT elicits 28--100% of hidden objective details across tasks, while 0% details are elicited in the baseline (GPT-4.1) or by the prefilled turn attack. SRFT-induced self-report is robust to adversarial decoy objectives for 3 out of 4 tasks. Error bars show 95% boostrapped CI.
• Figure 4: Effects of training the model to admit on-policy vs. off-policy false answers. SRFT trains models to make mistakes on-policy, then admit them. When fine-tuned only on admissions of off-policy, in-context mistakes (turn2-only), honesty generalization largely disappears. When we add a system instruction to generate false answers (turn2-only-sys-false), thereby making them more on-policy, honesty generalizations partly recover. We interrogate all models on the same GPT-4.1 trajectories (indicated by $\dagger$) to remove transcript variance. Error bars show 95% bootstrapped CIs.
• Figure 5: Effects of admissions being consistent with model belief vs. ground truth. On a training dataset containing hard questions the model consistently get wrong, aligning admissions with the ground truth (true admission) largely eliminates honesty generalization, whereas aligning them with the model’s belief (belief admission) substantially (but not fully) restores it. Error bars show 95% bootstrapped CIs.