A Systematic Evaluation of LLM Strategies for Mental Health Text Analysis: Fine-tuning vs. Prompt Engineering vs. RAG

TL;DR

The paper systematically evaluates three LLM deployment strategies—fine-tuning with LoRA, prompt engineering (zero-shot and few-shot), and retrieval-augmented generation (RAG)—for mental health text analysis using LLaMA 3-8B across the DAIR-AI Emotion and SWMH datasets. It demonstrates that fine-tuning yields the highest accuracy (up to 91% for emotion classification and 80% for mental health condition detection) but requires substantial computational resources, whereas prompt engineering and RAG offer more deployment flexibility with moderate performance. Zero-shot prompting emerges as a viable alternative for certain tasks (e.g., SWMH with 68% accuracy) and RAG shows improvements only when retrieval quality is high, highlighting trade-offs between accuracy, compute, and deployment practicality. The results provide practical guidance for implementing LLM-based mental health assessment tools and suggest avenues for hybrid methods and clinical validation to balance performance with real-world constraints.

Abstract

This study presents a systematic comparison of three approaches for the analysis of mental health text using large language models (LLMs): prompt engineering, retrieval augmented generation (RAG), and fine-tuning. Using LLaMA 3, we evaluate these approaches on emotion classification and mental health condition detection tasks across two datasets. Fine-tuning achieves the highest accuracy (91% for emotion classification, 80% for mental health conditions) but requires substantial computational resources and large training sets, while prompt engineering and RAG offer more flexible deployment with moderate performance (40-68% accuracy). Our findings provide practical insights for implementing LLM-based solutions in mental health applications, highlighting the trade-offs between accuracy, computational requirements, and deployment flexibility.

Figures (4)

• Figure 1: Overview of our experimental framework comparing three LLM deployment approaches (fine-tuning, prompt engineering, and RAG) for mental health text analysis. Each approach processes the same input data through distinct pipelines, enabling systematic comparison of their effectiveness.
• Figure 2: Architecture of our fine-tuning implementation, showing the integration of LoRA for parameter-efficient adaptation, the training process flow, and the evaluation pipeline.
• Figure 3: Architecture of our RAG model, illustrating the flow from input processing through retrieval and generation stages. The diagram shows how the system integrates embedded knowledge retrieval with LLM-based classification.
• Figure 4: Performance comparison across different approaches for both datasets. The graph shows accuracy scores for fine-tuning, zero-shot prompting, few-shot prompting, and RAG methods.