METIS: Mentoring Engine for Thoughtful Inquiry & Solutions

TL;DR

METIS, a tool-augmented, stage-aware assistant with literature search, curated guidelines, methodology checks, and memory, is built and evaluated against GPT-5 and Claude Sonnet 4.5 across six writing stages.

Abstract

Many students lack access to expert research mentorship. We ask whether an AI mentor can move undergraduates from an idea to a paper. We build METIS, a tool-augmented, stage-aware assistant with literature search, curated guidelines, methodology checks, and memory. We evaluate METIS against GPT-5 and Claude Sonnet 4.5 across six writing stages using LLM-as-a-judge pairwise preferences, student-persona rubrics, short multi-turn tutoring, and evidence/compliance checks. On 90 single-turn prompts, LLM judges preferred METIS to Claude Sonnet 4.5 in 71% and to GPT-5 in 54%. Student scores (clarity/actionability/constraint-fit; 90 prompts x 3 judges) are higher across stages. In multi-turn sessions (five scenarios/agent), METIS yields slightly higher final quality than GPT-5. Gains concentrate in document-grounded stages (D-F), consistent with stage-aware routing and groundings failure modes include premature tool routing, shallow grounding, and occasional stage misclassification.

Figures (5)

• Figure 1: METIS architecture. Stage detector and tool router select tools (Research Guidelines, web/document search, attachment search, methodology checks) based on writing stage. The agent synthesizes a reply and surfaces two self‑explanations (Intuition, Why this is principled), plus next steps and citations. Session memory maintains stage and constraints across turns.
• Figure 2: LLM-judge pairwise preferences across stages ($n{=}15$ prompts/stage; ties $\leq 8\%$ excluded). METIS wins $71\%$ vs Claude Sonnet 4.5 and $54\%$ vs GPT-5 overall; error bars show Wilson 95% CIs.
• Figure 3: LLM student-judge rubric trends across stages (A--F; 0--2 scale; mean $\pm$ 95% CI). METIS tracks above both baselines on student-perspective clarity, actionability, and constraint-fit; no expert judges are used for these scores.
• Figure 4: Multi-turn mentorship quality and a representative trajectory ($n{=}5$ scenarios/agent). Left: final LLM student-judge overall score (0--2; mean $\pm$ 95% CI). Right: per-scenario score trajectory for CivicTech NLP Volunteer with success markers; the horizontal line shows the success threshold. Success is scored post hoc at the first turn with overall $\geq 1.6$.
• Figure 5: Per-scenario multi-turn outcomes (faceted). Complements Figure \ref{['fig:multiturn']} by showing variability across scenarios; quality gains are consistent with modest efficiency differences.