Automatic Curriculum Expert Iteration for Reliable LLM Reasoning

TL;DR

This paper tackles hallucinations and laziness in LLM reasoning by introducing Auto-CEI, an automated curriculum framework that combines Expert Iteration with reward-shaped curricula to push the model toward longer, correct reasoning before admitting inability. It automatically estimates the reasoning capacity boundary via reasoning-step length, guiding when to respond assertively vs. defer to IDK. Through iterative expert-resampling and curriculum updates, Auto-CEI achieves higher precision and robust alignment across logical, mathematical, and planning tasks, outperforming several baselines. The work demonstrates strong potential for scalable, principled control of LLM reasoning behavior with practical impact on reliability and safety.

Abstract

Hallucinations (i.e., generating plausible but inaccurate content) and laziness (i.e. excessive refusals or defaulting to "I don't know") persist as major challenges in LLM reasoning. Current efforts to reduce hallucinations primarily focus on factual errors in knowledge-grounded tasks, often neglecting hallucinations related to faulty reasoning. Meanwhile, some approaches render LLMs overly conservative, limiting their problem-solving capabilities. To mitigate hallucination and laziness in reasoning tasks, we propose Automatic Curriculum Expert Iteration (Auto-CEI) to enhance LLM reasoning and align responses to the model's capabilities--assertively answering within its limits and declining when tasks exceed them. In our method, Expert Iteration explores the reasoning trajectories near the LLM policy, guiding incorrect paths back on track to reduce compounding errors and improve robustness; it also promotes appropriate "I don't know" responses after sufficient reasoning attempts. The curriculum automatically adjusts rewards, incentivizing extended reasoning before acknowledging incapability, thereby pushing the limits of LLM reasoning and aligning its behaviour with these limits. We compare Auto-CEI with various SOTA baselines across logical reasoning, mathematics, and planning tasks, where Auto-CEI achieves superior alignment by effectively balancing assertiveness and conservativeness. The code is available at https://github.com/SalesforceAIResearch/Auto-CEI .

Figures (10)

• Figure 1: Demonstration of the key idea of Auto-CEI. Auto-CEI automatically designs a sequence of curricula and uses Expert Iteration to train LLM in each curriculum. Expert Iteration explores the reasoning trajectories (i.e., Chain of thought) and learns to get incorrect paths back on track to reduce compounding errors. Using the reward (denoted by blue and red colour) produced by the curriculum, Expert Iteration also samples appropriate "I don’t know" responses (IDK responses) after sufficient reasoning attempts (in the blue region). To mitigate laziness, it avoids the short IDK responses (in the red region). The curriculum automatically adjusts rewards for the reasoning length, incentivizing extended reasoning before acknowledging incapability, thereby pushing the limits of LLM reasoning and aligning its assertive and conservative behaviours with these limits.
• Figure 2: The shape of $R(x, y)$ for $y$ is a refusal answer.
• Figure 3: The error and refusal rate (hallucination/IDK) according to the response length on different datasets.
• Figure 4: Correlation between reasoning length and labeled difficulty (MATH dataset). For each difficulty level, the mean reasoning length is plotted with error bars representing the standard error of the estimator.
• Figure 5: Histogram of reasoning lengths generated by Auto-CEI for the MATH dataset.