Fine-Tuning Small Language Models for Domain-Specific AI: An Edge AI Perspective

TL;DR

The paper tackles the challenge of running capable language models on edge devices by introducing the Shakti-SLM family (Shakti-100M, Shakti-250M, Shakti-500M). It combines efficient architectural decisions (RoPE, GQA, Block Sparse Attention, Sliding Window), quantization (int4/int5/int8) and domain-specific fine-tuning (SFT, RLHF or DPO) to enable on-device inference with strong general and domain-specific performance. Across general benchmarks and specialized domains such as healthcare, finance, and legal, Shakti models achieve competitive results while drastically reducing memory and energy demands, enabling deployment on smartphones, IoT devices, and edge GPUs. The work also emphasizes Responsible AI through on-device privacy, bias mitigation, and lower environmental impact, supported by responsible AI benchmarks. Collectively, Shakti demonstrates that compact models, when carefully engineered and aligned, can rival larger models in real-world edge scenarios and broaden access to private, low-latency AI.

Abstract

Deploying large scale language models on edge devices faces inherent challenges such as high computational demands, energy consumption, and potential data privacy risks. This paper introduces the Shakti Small Language Models (SLMs) Shakti-100M, Shakti-250M, and Shakti-500M which target these constraints headon. By combining efficient architectures, quantization techniques, and responsible AI principles, the Shakti series enables on-device intelligence for smartphones, smart appliances, IoT systems, and beyond. We provide comprehensive insights into their design philosophy, training pipelines, and benchmark performance on both general tasks (e.g., MMLU, Hellaswag) and specialized domains (healthcare, finance, and legal). Our findings illustrate that compact models, when carefully engineered and fine-tuned, can meet and often exceed expectations in real-world edge-AI scenarios.

Figures (6)

• Figure 1: Comparison results on academic benchmarks for Shakti-100M, Boomer-634Mboomer634m, SmolLM-135Mallal2024SmolLM, SmolLM-360Mallal2024SmolLM, and AMD-Llama-135Mamdllama135m, which are in the same parameter range.
• Figure 2: Comparison results on academic benchmarks for Shakti-250M, Boomer-1Bboomer1b, Boomer-634Mboomer634m, Qwen2.5-0.5Bqwen2.5, SmolLM-360Mallal2024SmolLM, and Llama 3.2 1Bllama3.2_1b.
• Figure 3: Comparison results on academic benchmarks for Shakti-500M, Boomer-1Bboomer1b, Boomer-634Mboomer634m, Qwen2.5-0.5Bqwen2.5, and Llama 3.2 1Bllama3.2_1b.
• Figure 4: Comparison results on medical and finance domain benchmarks for Shakti-250M, Phi-1.5-1.3Bphi15, Gemma-2Bgemma2b, and Opt-2.7Bzhang2022opt models, specifically for the Medical domain.
• Figure 5: Model size comparison before and after quantization. FP32 represents the original model size, while Q8, Q5, and Q4 represent increasingly aggressive quantization levels. Note the substantial reduction in memory footprint, with Q4 models requiring approximately 8x less memory than their FP32 counterparts.