RouteNator: A Router-Based Multi-Modal Architecture for Generating Synthetic Training Data for Function Calling LLMs

TL;DR

This work tackles the data scarcity and privacy constraints hindering fine-tuning LLMs for function calling by introducing a router-based, multi-modal synthetic data generation framework that integrates content metadata and domain knowledge graphs with text-to-text and vision-to-text models. By employing population-statistics routing across multiple data-generation routes, the approach produces diverse, realistic training data that better matches real user distributions. Evaluations on a golden dataset show significant improvements in function-call classification and API-parameter selection, with the router-based data outperforming traditional generation methods and enabling effective fine-tuning of Gorilla OpenFunctions v2 and several small language models. The architecture demonstrates substantial practical impact for domain-specific function calling and lays groundwork for broader, multilingual, and multi-domain extension.

Abstract

This paper addresses fine-tuning Large Language Models (LLMs) for function calling tasks when real user interaction data is unavailable. In digital content creation tools, where users express their needs through natural language queries that must be mapped to API calls, the lack of real-world task-specific data and privacy constraints for training on it necessitate synthetic data generation. Existing approaches to synthetic data generation fall short in diversity and complexity, failing to replicate real-world data distributions and leading to suboptimal performance after LLM fine-tuning. We present a novel router-based architecture that leverages domain resources like content metadata and structured knowledge graphs, along with text-to-text and vision-to-text language models to generate high-quality synthetic training data. Our architecture's flexible routing mechanism enables synthetic data generation that matches observed real-world distributions, addressing a fundamental limitation of traditional approaches. Evaluation on a comprehensive set of real user queries demonstrates significant improvements in both function classification accuracy and API parameter selection. Models fine-tuned with our synthetic data consistently outperform traditional approaches, establishing new benchmarks for function calling tasks.

Figures (9)

• Figure 1: Data generation architecture overview integrating metadata, knowledge graph, and visual content. A "Weighted Router" directs text and image inputs to different prompt categories: length-based, API-based, and media type. They are processed by Text-to-Text and Vision-to-Text LLMs to generate synthetic data for downstream tasks.
• Figure 2: Knowledge Graph of concepts linked by edges
• Figure 3: Comparison of word count distribution (Mean, Median and Interquartile Range) across the real and synthetically generated datasets (Heuristic, Single Prompt and Router)
• Figure 4: Comparison of Content Type distribution
• Figure 5: Comparison of normalized keyword positions