Lost in Execution: On the Multilingual Robustness of Tool Calling in Large Language Models

TL;DR

This work tackles multilingual robustness in tool calling by introducing MLCL, a diagnostic benchmark built on BFCL to study how non-English input affects execution at the language–execution boundary. It reveals parameter value language mismatch as the dominant failure mode, with semantic understanding often intact across Chinese, Hindi, and Igbo. The study shows that simple inference-time strategies (e.g., translation, explicit prompting) reduce certain errors but fail to reach English-level performance, highlighting a system-level challenge in aligning language with execution interfaces. The findings underscore the need for execution-aware design and robust multilingual tool-calling interfaces to enable globally deployed LLM agents.

Abstract

Large Language Models (LLMs) are increasingly deployed as agents that invoke external tools through structured function calls. While recent work reports strong tool-calling performance under standard English-centric evaluations, the robustness of tool calling under multilingual user interactions remains underexplored. In this work, we introduce MLCL, a diagnostic benchmark, and conduct a systematic evaluation of multilingual tool calling across Chinese, Hindi, and the low-resource language Igbo. Through fine-grained error analysis, we show that many failures occur despite correct intent understanding and tool selection. We identify parameter value language mismatch as a dominant failure mode, where models generate semantically appropriate parameter values in the user's language, violating language-invariant execution conventions. We further evaluate several inference-time system strategies and find that while these strategies substantially reduce language-induced execution errors, none of them can fully recover English-level performance.

Figures (14)

• Figure 1: Multilingual tool-calling failures stem from execution-level parameter mismatches. The same user intent expressed in English, Chinese, Hindi, and Igbo yields semantically appropriate tool calls that become non-executable when parameter values violate English-only execution conventions, a failure mode we refer to as parameter value language mismatch.
• Figure 2: Diagnostic design of the Multilingual Tool-Calling (MLCL) Benchmark and inference-time mitigation strategies. The benchmark systematically varies query language composition (NT, PAR, FT) and semantic perturbations (NO, PARA, SYNO) to isolate multilingual execution failures under a fixed, language-invariant tool interface. This design exposes cases where tool calls are semantically correct but operationally invalid due to execution-level violations, such as parameter value language mismatch. The right panel summarizes inference-time mitigation strategies (PT, PRE, POST) evaluated in this work, which reduce some multilingual errors but do not fully eliminate execution-level gaps.
• Figure 3: Error taxonomy and severity levels used for evaluating multilingual tool calling. Error categories are ordered from most severe (top) to least severe (bottom), with illustrative examples for each category.
• Figure 4: Error distributions for five representative models under English (NT), partially translated (PAR), and fully translated (FT) queries in Chinese, Hindi, and Igbo. Across languages and model families, moving from NT to FT systematically increases execution-level errors, driven primarily by parameter value language mismatch (in purple color), while PAR substantially reduces these violations. The consistency of this trend across models indicates a shared failure mechanism at the language--execution interface rather than model-specific weaknesses.
• Figure 5: Evaluation of semantic perturbations for five representative models on the Chinese dataset under paraphrasing (PARA) and synonym substitution (SYNO) across NT, PAR, and FT settings. Semantic perturbations substantially increase errors when exact English parameter surface forms are required (NT), but have limited additional impact in fully translated (FT) queries dominated by parameter value language mismatch (in purple color). The partially translated (PAR) setting exhibits intermediate sensitivity, indicating an interaction between semantic variation and execution-level language constraints.