Evolution of meta's llama models and parameter-efficient fine-tuning of large language models: a survey

TL;DR

This survey analyzes Meta AI's LLaMA lineage (from LLaMA 1 to LLaMA 4) and the associated parameter-efficient fine-tuning techniques that enable practical deployment of very large models. It details five PEFT methods—LoRA, LLaMA-Adapter (V1/V2), LLaMA-Excitor, and QLoRA—along with architectural traits such as sparse Mixture-of-Experts and multimodal capabilities, and it assesses their effects on training cost, parameter counts, and downstream performance. The paper combines model-evolution insights with real-world domain applications (NLP, healthcare, legal, vision-language tasks) and provides a structured meta-analysis comparing methods. It highlights the trade-offs between parameter efficiency, accuracy, and multimodal capacity, and discusses challenges in hardware, optimization stability, and language coverage. The findings indicate that PEFT approaches can closely match full fine-tuning performance with orders of magnitude fewer trainable parameters, enabling scalable, domain-specific AI while underscoring the need for robust evaluation and responsible deployment.

Abstract

This review surveys the rapid evolution of Meta AI's LLaMA (Large Language Model Meta AI) series - from LLaMA 1 through LLaMA 4 and the specialized parameter-efficient fine-tuning (PEFT) methods developed for these models. We first describe the LLaMA family of foundation models (7B-65B to 288B parameters), their architectures (including native multimodal and Mixtureof-Experts variants), and key performance characteristics. We then describe and discuss the concept of PEFT, which adapts large pre-trained models by updating only a small subset of parameters, and review five PEFT methods that have been applied to LLaMA: LoRA (Low-Rank Adaptation), LLaMA-Adapter V1 and V2, LLaMA-Excitor, and QLoRA (Quantized LoRA). We discuss each method's mechanism, parameter savings, and example application to LLaMA (e.g., instruction tuning, multimodal tasks). We provide structured discussion and analysis of model and adapter architectures, parameter counts, and benchmark results (including examples where fine-tuned LLaMA models outperform larger baselines). Finally, we examine real-world use cases where LLaMA-based models and PEFT have been successfully applied (e.g., legal and medical domains), and we discuss ongoing challenges and future research directions (such as scaling to even larger contexts and improving robustness). This survey paper provides a one-stop resource for ML researchers and practitioners interested in LLaMA models and efficient fine-tuning strategies.

Figures (14)

• Figure 1: Evolution of Meta's LLaMA models from foundation models (7B-65B) through chat-optimized variants to sparse Mixture-of-Experts (MoE) architectures
• Figure 2: Flowchart of the Survey Structure for LLaMA and Parameter-Efficient Fine-Tuning Methods
• Figure 3: LLaMA Model Scaling Timeline: From 7B to Trillions (2023-2025)
• Figure 4: Overview of the Training Pipeline Combining SFT and DPO
• Figure 5: Sparse Mixture-of-Experts Transformer block