Answering Students' Questions on Course Forums Using Multiple Chain-of-Thought Reasoning and Finetuning RAG-Enabled LLM

TL;DR

The paper tackles the challenge of answering students' questions on course forums at scale by developing a retrieval-augmented QA system that leverages a local course knowledge base. It combines open-source LLMs with RAG and fine-tuning (via LoRA) to produce context-aware answers, while employing multi-chain-of-thought reasoning to reduce hallucination and improve transparency. Experiments on HotpotQA demonstrate that RAG with fine-tuning substantially boosts performance (e.g., F1 improvements from 19.0 to 62.2 when using finetuned Llama-3.2-3B-Instruct with RAG), and that pairing multiple chain-of-thoughts with evidence leads to optimal results at two chains. The work highlights practical benefits for education, including 24/7 responsiveness and reduced instructor workload, and points to future directions such as multimodal retrieval and human-in-the-loop enhancements to further improve reliability and learning outcomes.

Abstract

The course forums are increasingly significant and play vital role in facilitating student discussions and answering their questions related to the course. It provides a platform for students to post their questions related to the content and admin issues related to the course. However, there are several challenges due to the increase in the number of students enrolled in the course. The primary challenge is that students' queries cannot be responded immediately and the instructors have to face lots of repetitive questions. To mitigate these issues, we propose a question answering system based on large language model with retrieval augmented generation (RAG) method. This work focuses on designing a question answering system with open source Large Language Model (LLM) and fine-tuning it on the relevant course dataset. To further improve the performance, we use a local knowledge base and applied RAG method to retrieve relevant documents relevant to students' queries, where the local knowledge base contains all the course content. To mitigate the hallucination of LLMs, We also integrate it with multi chain-of-thought reasoning to overcome the challenge of hallucination in LLMs. In this work, we experiment fine-tuned LLM with RAG method on the HotpotQA dataset. The experimental results demonstrate that the fine-tuned LLM with RAG method has a strong performance on question answering task.

Figures (3)

• Figure 1: Structure of RAG system
• Figure 2: Multi-chains Reasoning with RAG VS Traditional Question-answering System
• Figure 3: Example of Multi-chains Reasoning.