GUIDE: Towards Scalable Advising for Research Ideas

TL;DR

GUIDE presents a scalable, retrieval-augmented framework for hypothesis evaluation and experimental design guidance. By compressing literature into modular summaries and enforcing rubric-guided alignment via RAFT, a relatively small model (GUIDE-7B) outperforms larger general-purpose LLMs on ICLR 2025 acceptance predictions, especially when predictions are high-confidence. The approach combines four components—Guidelines, Understanding, Information Retrieval, Direction, and Explanation—into a structured feedback loop that reduces hallucinations and improves actionable critique. Empirical results show strong Top-30% precision and high-confidence performance, with ablations demonstrating the value of modular summarization, rubrics, and uncertainty-aware selection. The work suggests a promising path for scalable, domain-aware AI advising systems that can meaningfully augment scientific ideation and decision-making.

Abstract

The field of AI research is advancing at an unprecedented pace, enabling automated hypothesis generation and experimental design across diverse domains such as biology, mathematics, and artificial intelligence. Despite these advancements, there remains a significant gap in the availability of scalable advising systems capable of providing high-quality, well-reasoned feedback to refine proposed hypotheses and experimental designs. To address this challenge, we explore key factors that underlie the development of robust advising systems, including model size, context length, confidence estimation, and structured reasoning processes. Our findings reveal that a relatively small model, when equipped with a well-compressed literature database and a structured reasoning framework, can outperform powerful general-purpose language models such as Deepseek-R1 in terms of acceptance rates for self-ranked top-30% submissions to ICLR 2025. Moreover, when limited to high-confidence predictions, our system achieves an acceptance rate exceeding 90% on the ICLR 2025 test set, underscoring its potential to significantly enhance the quality and efficiency of hypothesis generation and experimental design. The code is released at https://github.com/HowardLiu0830/GUIDE-Research-Idea-Evaluation.

Figures (7)

• Figure 1: Contribution extraction with learned prompts.
• Figure 2: GUIDE: a RAG-based Advising System. The GUIDE pipeline begins by receiving the target paper's abstract, contribution, method, and experiment setup. For each of these sections, it retrieves corresponding content from a database of prior works. These target–exemplar pairs are fed into an LLM, which applies predefined rubrics to generate a structured advice. Finally, the idea's abstract and contribution together with this structured advice are passed to a lightweight classifier that produces the final numerical rating.
• Figure 3: Training Pipeline: Overview of the two-stage training process for GUIDE-7B. Stage 1 (Warming Up) initializes GUIDE-7B with 4K idea-evaluation examples distilled from DeepSeek-R1, pairing each idea with rubric-based advice. Stage 2 (RAFT) further aligns GUIDE-7B with human evaluations by optimizing advice similarity (via ROUGE) and rating similarity (dot product of predicted and actual rating distributions). In RAFT, GUIDE-7B generates and selects the top-$k$ candidate advice responses for additional fine-tuning. Low-rated ideas encourage identification of weaknesses, while high-rated ideas prompt more positive feedback.
• Figure 4: Uncertainty Analysis: Precision means the ratio of actually accepted papers over all papers that were within the specific confidence and predicted rating ranking interval. Predicted ranking interval means the set of papers sorted in descending order in terms of predicted rating.
• Figure 5: Distribution of ratings for human reviewers and various LLMs. General-purpose LLMs show optimism bias with ratings skewed toward higher intervals, while GUIDE-7B yields a more balanced, human-like rating distribution.