TinyLLM: Evaluation and Optimization of Small Language Models for Agentic Tasks on Edge Devices

TL;DR

This work evaluates small language models (sub-3B) for agentic tool/API calling on edge devices using the Berkeley Function Calling Leaderboard (BFCL). It contrasts conventional fine-tuning, reinforcement learning, and Direct Preference Optimization (DPO), proposing a DPO pipeline with AgentBank/ALFRED data and hybridization to balance accuracy, latency, and privacy. The results show medium-scale SLMs (1-3B) outperform ultra-compact variants, achieving up to 65.74% overall BFCL accuracy and strong non-live/AST performance, with multi-turn capabilities up to 55.62% for the best model. The study offers a practical roadmap for private, low-latency autonomous agents on edge devices, emphasizing hybrid optimization, diverse evaluation, and ongoing data-centric alignment.

Abstract

This paper investigates the effectiveness of small language models (SLMs) for agentic tasks (function/tool/API calling) with a focus on running agents on edge devices without reliance on cloud infrastructure. We evaluate SLMs using the Berkeley Function Calling Leaderboard (BFCL) framework and describe parameter-driven optimization strategies that include supervised fine-tuning (SFT), parameter-efficient fine-tuning (PEFT), reinforcement learning (RL)-based optimization, preference alignment via Direct Preference Optimization (DPO), and hybrid methods. We report results for models including TinyAgent, TinyLlama, Qwen, and xLAM across BFCL categories (simple, multiple, parallel, parallel-multiple, and relevance detection), both in live and non-live settings, and in multi-turn evaluations. We additionally detail a DPO training pipeline constructed from AgentBank data (e.g., ALFRED), including our conversion of SFT data to chosen-rejected pairs using TinyLlama responses as rejected outputs and manual validation. Our results demonstrate clear accuracy differences across model scales where medium-sized models (1-3B parameters) significantly outperform ultra-compact models (<1B parameters), achieving up to 65.74% overall accuracy, and 55.62% multi-turn accuracy with hybrid optimization. This study highlights the importance of hybrid optimization strategies that enable small language models to deliver accurate, efficient, and stable agentic AI on edge devices, making privacy-preserving, low-latency autonomous agents practical beyond the cloud.

Figures (3)

• Figure 1: Proposed Pipeline of TinyLLM
• Figure 2: Evaluation dataset's data distribution
• Figure 3: Direct Preference Optimization Pipeline