Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Small Models, Big Tasks: An Exploratory Empirical Study on Small Language Models for Function Calling

Ishan Kavathekar, Raghav Donakanti, Ponnurangam Kumaraguru, Karthik Vaidhyanathan

TL;DR

This paper evaluates Small Language Models (SLMs) for function calling across zero-shot, few-shot, and LoRA-based finetuning settings, including edge-device deployment with GGUF quantization and prompt-injection robustness. It finds that zero-shot performance is generally poor, few-shot prompts yield notable improvements for some models, and finetuning yields the strongest gains, though many models still struggle with strict output formats. Edge deployment reveals significant latency and memory challenges, with performance not consistently aligning with server results, highlighting the need for constrained decoding, validators, and hybrid architectures. The authors provide finetuned models and a replication package, offering practical guidance and avenues for future work in robust, on-device function calling and cross-domain generalization.

Abstract

Function calling is a complex task with widespread applications in domains such as information retrieval, software engineering and automation. For example, a query to book the shortest flight from New York to London on January 15 requires identifying the correct parameters to generate accurate function calls. Large Language Models (LLMs) can automate this process but are computationally expensive and impractical in resource-constrained settings. In contrast, Small Language Models (SLMs) can operate efficiently, offering faster response times, and lower computational demands, making them potential candidates for function calling on edge devices. In this exploratory empirical study, we evaluate the efficacy of SLMs in generating function calls across diverse domains using zero-shot, few-shot, and fine-tuning approaches, both with and without prompt injection, while also providing the finetuned models to facilitate future applications. Furthermore, we analyze the model responses across a range of metrics, capturing various aspects of function call generation. Additionally, we perform experiments on an edge device to evaluate their performance in terms of latency and memory usage, providing useful insights into their practical applicability. Our findings show that while SLMs improve from zero-shot to few-shot and perform best with fine-tuning, they struggle significantly with adhering to the given output format. Prompt injection experiments further indicate that the models are generally robust and exhibit only a slight decline in performance. While SLMs demonstrate potential for the function call generation task, our results also highlight areas that need further refinement for real-time functioning.

Small Models, Big Tasks: An Exploratory Empirical Study on Small Language Models for Function Calling

TL;DR

This paper evaluates Small Language Models (SLMs) for function calling across zero-shot, few-shot, and LoRA-based finetuning settings, including edge-device deployment with GGUF quantization and prompt-injection robustness. It finds that zero-shot performance is generally poor, few-shot prompts yield notable improvements for some models, and finetuning yields the strongest gains, though many models still struggle with strict output formats. Edge deployment reveals significant latency and memory challenges, with performance not consistently aligning with server results, highlighting the need for constrained decoding, validators, and hybrid architectures. The authors provide finetuned models and a replication package, offering practical guidance and avenues for future work in robust, on-device function calling and cross-domain generalization.

Abstract

Function calling is a complex task with widespread applications in domains such as information retrieval, software engineering and automation. For example, a query to book the shortest flight from New York to London on January 15 requires identifying the correct parameters to generate accurate function calls. Large Language Models (LLMs) can automate this process but are computationally expensive and impractical in resource-constrained settings. In contrast, Small Language Models (SLMs) can operate efficiently, offering faster response times, and lower computational demands, making them potential candidates for function calling on edge devices. In this exploratory empirical study, we evaluate the efficacy of SLMs in generating function calls across diverse domains using zero-shot, few-shot, and fine-tuning approaches, both with and without prompt injection, while also providing the finetuned models to facilitate future applications. Furthermore, we analyze the model responses across a range of metrics, capturing various aspects of function call generation. Additionally, we perform experiments on an edge device to evaluate their performance in terms of latency and memory usage, providing useful insights into their practical applicability. Our findings show that while SLMs improve from zero-shot to few-shot and perform best with fine-tuning, they struggle significantly with adhering to the given output format. Prompt injection experiments further indicate that the models are generally robust and exhibit only a slight decline in performance. While SLMs demonstrate potential for the function call generation task, our results also highlight areas that need further refinement for real-time functioning.
Paper Structure (17 sections, 3 equations, 4 figures, 5 tables)

This paper contains 17 sections, 3 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Small Language Models (SLMs)
  4. Zero-shot, Few-shot, and Finetuning approaches for inference
  5. GGUF models
  6. Prompt Injection
  7. Related Work
  8. Study Design
  9. Goal
  10. Research Questions
  11. Experiment Subject
  12. Experimental Procedure
  13. Metrics
  14. Results
  15. Discussion
  16. ...and 2 more sections

Figures (4)

  • Figure 1: Overview of the study design illustrating different components of the research methodology.
  • Figure 2: Sample datapoint from the Salesforce-XLAM function-calling dataset, illustrating its components: user query, available tools, and the answer.
  • Figure 3: Responses of Deepseek-Coder and Phi-3-mini across settings highlighting incorrect (red) and correct (green) outputs.
  • Figure 4: Comparison of latency (in sec) and Memory usage (in MB) of Models on edge and server environments.