Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Assessing Personalized AI Mentoring with Large Language Models in the Computing Field

Xiao Luo, Sean O'Connell, Shamima Mithun

TL;DR

This study assesses three Large Language Models (GPT-4o, LLaMA 3, PaLM 2) for personalized career mentoring in computing using zero-shot prompts and three diverse student profiles. An NLP analytics pipeline evaluates how well responses reflect mentees' social and educational backgrounds, complemented by a human survey to gauge usefulness and personalization. Findings indicate GPT-4o generally delivers more personalized and useful mentoring, while LLaMA 3 shows stronger profile-based tailoring and PaLM 2 sits between. The work underscores the importance of human-in-the-loop mentoring and prompts careful consideration of ethics, privacy, and bias when deploying LLM-based mentoring tools at scale.

Abstract

This paper provides an in-depth evaluation of three state-of-the-art Large Language Models (LLMs) for personalized career mentoring in the computing field, using three distinct student profiles that consider gender, race, and professional levels. We evaluated the performance of GPT-4, LLaMA 3, and Palm 2 using a zero-shot learning approach without human intervention. A quantitative evaluation was conducted through a custom natural language processing analytics pipeline to highlight the uniqueness of the responses and to identify words reflecting each student's profile, including race, gender, or professional level. The analysis of frequently used words in the responses indicates that GPT-4 offers more personalized mentoring compared to the other two LLMs. Additionally, a qualitative evaluation was performed to see if human experts reached similar conclusions. The analysis of survey responses shows that GPT-4 outperformed the other two LLMs in delivering more accurate and useful mentoring while addressing specific challenges with encouragement languages. Our work establishes a foundation for developing personalized mentoring tools based on LLMs, incorporating human mentors in the process to deliver a more impactful and tailored mentoring experience.

Assessing Personalized AI Mentoring with Large Language Models in the Computing Field

TL;DR

This study assesses three Large Language Models (GPT-4o, LLaMA 3, PaLM 2) for personalized career mentoring in computing using zero-shot prompts and three diverse student profiles. An NLP analytics pipeline evaluates how well responses reflect mentees' social and educational backgrounds, complemented by a human survey to gauge usefulness and personalization. Findings indicate GPT-4o generally delivers more personalized and useful mentoring, while LLaMA 3 shows stronger profile-based tailoring and PaLM 2 sits between. The work underscores the importance of human-in-the-loop mentoring and prompts careful consideration of ethics, privacy, and bias when deploying LLM-based mentoring tools at scale.

Abstract

This paper provides an in-depth evaluation of three state-of-the-art Large Language Models (LLMs) for personalized career mentoring in the computing field, using three distinct student profiles that consider gender, race, and professional levels. We evaluated the performance of GPT-4, LLaMA 3, and Palm 2 using a zero-shot learning approach without human intervention. A quantitative evaluation was conducted through a custom natural language processing analytics pipeline to highlight the uniqueness of the responses and to identify words reflecting each student's profile, including race, gender, or professional level. The analysis of frequently used words in the responses indicates that GPT-4 offers more personalized mentoring compared to the other two LLMs. Additionally, a qualitative evaluation was performed to see if human experts reached similar conclusions. The analysis of survey responses shows that GPT-4 outperformed the other two LLMs in delivering more accurate and useful mentoring while addressing specific challenges with encouragement languages. Our work establishes a foundation for developing personalized mentoring tools based on LLMs, incorporating human mentors in the process to deliver a more impactful and tailored mentoring experience.

Paper Structure

This paper contains 13 sections, 1 equation, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLMs in Education
  4. LLMs for Mentoring
  5. Methodology
  6. Student Profile and Mentoring Questions
  7. AI Mentoring using LLMs
  8. NLP Analysis Pipeline
  9. Design of Human Evaluation
  10. Experimental Settings and Evaluation Results
  11. Result of NLP Analysis
  12. Result of Human Evaluation
  13. Conclusion and Future work

Figures (6)

  • Figure 1: Overview of the AI mentoring and analysis using LLMs
  • Figure 2: Design of the prompt for AI mentoring
  • Figure 3: Human Evaluation of the LLMs towards Mentoring
  • Figure 4: Word Clouds of All question answers for profile M-AA-J-CS, M-W-F-U, and F-H-F-CS using GPT-4o
  • Figure 5: Word Clouds of All question answers for profile M-AA-J-CS, M-W-F-U, and F-H-F-CS using LLaMA 3
  • ...and 1 more figures