Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Arabic Prompts with English Tools: A Benchmark

Konstantin Kubrak, Ahmed El-Moselhy, Ammar Alsulami, Remaz Altuwaim, Hassan Ismail Fawaz, Faisal Alsaby

TL;DR

The paper addresses the lack of Arabic tool-calling benchmarks by adapting the Berkeley Function Calling Leaderboard to Arabic, enabling evaluation of Arabic agentic workflows in LLMs. It translates BFCL components into a multilingual corpus and tests five open-source models (GPT-OSS-20b, Llama-3.3-70b, Qwen3 variants) across a 16-configuration matrix to isolate the effects of language in prompts and tool descriptions. The results reveal a pronounced performance gap when prompts are in Arabic, including a compounding penalty when both user queries and tool descriptions are in Arabic, indicating English-centric internal representations for tool-use. The findings underscore the need for native Arabic benchmarks, multilingual fine-tuning, and Arabic-aware evaluation metrics to develop robust and linguistically equitable Arabic AI agents.

Abstract

Large Language Models (LLMs) are now integral to numerous industries, increasingly serving as the core reasoning engine for autonomous agents that perform complex tasks through tool-use. While the development of Arabic-native LLMs is accelerating, the benchmarks for evaluating their capabilities lag behind, with most existing frameworks focusing on English. A critical and overlooked area is tool-calling, where the performance of models prompted in non-English languages like Arabic is poorly understood, especially since these models are often pretrained on predominantly English data. This paper addresses this critical gap by introducing the first dedicated benchmark for evaluating the tool-calling and agentic capabilities of LLMs in the Arabic language. Our work provides a standardized framework to measure the functional accuracy and robustness of models in Arabic agentic workflows. Our findings reveal a huge performance gap: when users interact in Arabic, tool-calling accuracy drops by an average of 5-10\%, regardless of whether the tool descriptions themselves are in Arabic or English. By shedding light on these critical challenges, this benchmark aims to foster the development of more reliable and linguistically equitable AI agents for Arabic-speaking users.

Arabic Prompts with English Tools: A Benchmark

TL;DR

The paper addresses the lack of Arabic tool-calling benchmarks by adapting the Berkeley Function Calling Leaderboard to Arabic, enabling evaluation of Arabic agentic workflows in LLMs. It translates BFCL components into a multilingual corpus and tests five open-source models (GPT-OSS-20b, Llama-3.3-70b, Qwen3 variants) across a 16-configuration matrix to isolate the effects of language in prompts and tool descriptions. The results reveal a pronounced performance gap when prompts are in Arabic, including a compounding penalty when both user queries and tool descriptions are in Arabic, indicating English-centric internal representations for tool-use. The findings underscore the need for native Arabic benchmarks, multilingual fine-tuning, and Arabic-aware evaluation metrics to develop robust and linguistically equitable Arabic AI agents.

Abstract

Large Language Models (LLMs) are now integral to numerous industries, increasingly serving as the core reasoning engine for autonomous agents that perform complex tasks through tool-use. While the development of Arabic-native LLMs is accelerating, the benchmarks for evaluating their capabilities lag behind, with most existing frameworks focusing on English. A critical and overlooked area is tool-calling, where the performance of models prompted in non-English languages like Arabic is poorly understood, especially since these models are often pretrained on predominantly English data. This paper addresses this critical gap by introducing the first dedicated benchmark for evaluating the tool-calling and agentic capabilities of LLMs in the Arabic language. Our work provides a standardized framework to measure the functional accuracy and robustness of models in Arabic agentic workflows. Our findings reveal a huge performance gap: when users interact in Arabic, tool-calling accuracy drops by an average of 5-10\%, regardless of whether the tool descriptions themselves are in Arabic or English. By shedding light on these critical challenges, this benchmark aims to foster the development of more reliable and linguistically equitable AI agents for Arabic-speaking users.
Paper Structure (41 sections, 10 figures, 2 tables)

This paper contains 41 sections, 10 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Dataset Translation and Preparation
  5. Models Under Evaluation
  6. GPT-OSS-20b
  7. Llama-3.3-70b
  8. Qwen3
  9. Experimental Design
  10. System Prompt Language
  11. Invocation Style
  12. User Query Language
  13. Function Description Language
  14. Task Categories
  15. Irrelevance Detection
  16. ...and 26 more sections

Figures (10)

  • Figure 1: Accuracy comparison between llama-3.3-70b (x-axis) and qwen3-30b (y-axis).
  • Figure 2: Accuracy comparison between llama-3.3-70b (x-axis) and qwen3-4b (y-axis).
  • Figure 3: Accuracy comparison between qwen3-30b (x-axis) and qwen3-4b (y-axis). Note the 'x' markers for prompt-based invocation.
  • Figure 4: Accuracy comparison between gpt-oss-20b (x-axis) and qwen3-4b (y-axis).
  • Figure 5: Accuracy comparison between qwen3-4b (x-axis) and qwen3-4b-thinking (y-axis).
  • ...and 5 more figures