AdvisingWise: Supporting Academic Advising in Higher Education Settings Through a Human-in-the-Loop Multi-Agent Framework

TL;DR

AdvisingWise presents a human-in-the-loop, three-phase multi-agent framework for academic advising that automates information retrieval and draft generation while ensuring advisor validation. By integrating adaptive student profiles and authoritative institutional sources within a ReAct-style information collection process, it achieves reliable, personalized guidance. Mixed-methods evaluation shows high expert-validated accuracy, superior retrieval performance against a RAG baseline, and positive shifts in advisors' attitudes toward AI integration. The work demonstrates how human-AI collaboration can reduce workload in advising without compromising personal connection and policy fidelity, with implications for scalable, context-aware advising in higher education.

Abstract

Academic advising is critical to student success in higher education, yet high student-to-advisor ratios limit advisors' capacity to provide timely support, particularly during peak periods. Recent advances in Large Language Models (LLMs) present opportunities to enhance the advising process. We present AdvisingWise, a multi-agent system that automates time-consuming tasks, such as information retrieval and response drafting, while preserving human oversight. AdvisingWise leverages authoritative institutional resources and adaptively prompts students about their academic backgrounds to generate reliable, personalized responses. All system responses undergo human advisor validation before delivery to students. We evaluate AdvisingWise through a mixed-methods approach: (1) expert evaluation on responses of 20 sample queries, (2) LLM-as-a-judge evaluation of the information retrieval strategy, and (3) a user study with 8 academic advisors to assess the system's practical utility. Our evaluation shows that AdvisingWise produces accurate, personalized responses. Advisors reported increasingly positive perceptions after using AdvisingWise, as their initial concerns about reliability and personalization diminished. We conclude by discussing the implications of human-AI synergy on the practice of academic advising.

Figures (6)

• Figure 1: Three-phase multi-agent architecture. Phase 1 (Query Preprocessing): Academic Info Extractor, Query Rewriter, and Query Classifier—if classified as off-topic, the query bypasses Phases 2-3. Phase 2 (Information Collection): Thought and Action agents collaborate iteratively (up to 4 cycles) to gather information from institutional knowledge base, course database, web sources, or student prompts. Phase 3 (Response Generation): Answer Generator and Quality Controller produce draft outputs. Red-highlighted components (detailed answer, summary, cited sources, and advisor notes) appear in the final draft response delivered to advisors.
• Figure 2: Advisor interface: (left) notification of student question, (right) AI-generated draft with response, summary, sources. Student names are pseudonyms.
• Figure 3: Example draft responses demonstrating key features: (left) advisor-only notes flag uncertainty, (right) personalized response based on student academic profile, which is explicitly cited as a source
• Figure 4: Student interaction flow: AdvisingWise prompts for academic background before generating personalized draft for advisor review.
• Figure 5: Perceived Usefulness of Using LLM-Powered Chatbots in Advising