Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FABRIC: Framework for Agent-Based Realistic Intelligence Creation

Abhigya Verma, Seganrasan Subramanian, Nandhakumar Kandasamy, Naman Gupta

TL;DR

FABRIC presents a fully synthetic, LLM-only pipeline suite for generating agentic data that couples user intent, tool calls, and execution traces under strict schema constraints. The framework uses four modular pipelines (RecordSynth, DAGFirstGeneration, MultiTurnDialogueSynth, AgenticRecordRollout) guided by declarative DAG templates to produce end-to-end, machine-checkable supervision signals across single-turn and multi-turn contexts. A key contribution is the integration of JSON-schema validation and judge-based filtering to ensure semantic and syntactic fidelity, enabling scalable, reproducible data generation for tool-use training and evaluation. The work aims to close the gap between free-form language data and executable, tool-enabled agent behavior, accelerating the development of robust, tool-using LLMs with scalable benchmarking and training datasets.

Abstract

Large language models (LLMs) are increasingly deployed as agents, expected to decompose goals, invoke tools, and verify results in dynamic environments. Realizing these capabilities requires access to agentic data-structured interaction records that couple user intents with tool specifications, argument-grounded calls, and verifiable execution traces. However, collecting such data from human annotators is costly, time-consuming, and difficult to scale. We present a unified framework for synthesizing agentic data using only LLMs, without any human-in-the-loop supervision. This framework decomposes generation into modular pipelines that produce complete interaction records spanning task specifications, tool definitions, policy pseudocode, natural language exchanges, and execution traces. Records conform to strict syntactic and semantic constraints, ensuring machine-parseability and faithful alignment across inputs, outputs, and tool calls. Beyond single tasks, there is support for both multi-task and multi-turn agent interactions, enabling the construction of datasets that reflect the full spectrum of tool-use competencies. To ensure quality and consistency, the framework integrates constrained generation formats, JSON-schema validation, and judge-based filtering. This paper formalizes the schema for agentic records, details the prompt design principles that guide generation, and introduces scalable pipelines for high-quality synthetic data. By providing a reproducible, LLM-only alternative to manual collection, hence advancing the development of agentic LLMs capable of robust tool use.

FABRIC: Framework for Agent-Based Realistic Intelligence Creation

TL;DR

FABRIC presents a fully synthetic, LLM-only pipeline suite for generating agentic data that couples user intent, tool calls, and execution traces under strict schema constraints. The framework uses four modular pipelines (RecordSynth, DAGFirstGeneration, MultiTurnDialogueSynth, AgenticRecordRollout) guided by declarative DAG templates to produce end-to-end, machine-checkable supervision signals across single-turn and multi-turn contexts. A key contribution is the integration of JSON-schema validation and judge-based filtering to ensure semantic and syntactic fidelity, enabling scalable, reproducible data generation for tool-use training and evaluation. The work aims to close the gap between free-form language data and executable, tool-enabled agent behavior, accelerating the development of robust, tool-using LLMs with scalable benchmarking and training datasets.

Abstract

Large language models (LLMs) are increasingly deployed as agents, expected to decompose goals, invoke tools, and verify results in dynamic environments. Realizing these capabilities requires access to agentic data-structured interaction records that couple user intents with tool specifications, argument-grounded calls, and verifiable execution traces. However, collecting such data from human annotators is costly, time-consuming, and difficult to scale. We present a unified framework for synthesizing agentic data using only LLMs, without any human-in-the-loop supervision. This framework decomposes generation into modular pipelines that produce complete interaction records spanning task specifications, tool definitions, policy pseudocode, natural language exchanges, and execution traces. Records conform to strict syntactic and semantic constraints, ensuring machine-parseability and faithful alignment across inputs, outputs, and tool calls. Beyond single tasks, there is support for both multi-task and multi-turn agent interactions, enabling the construction of datasets that reflect the full spectrum of tool-use competencies. To ensure quality and consistency, the framework integrates constrained generation formats, JSON-schema validation, and judge-based filtering. This paper formalizes the schema for agentic records, details the prompt design principles that guide generation, and introduces scalable pipelines for high-quality synthetic data. By providing a reproducible, LLM-only alternative to manual collection, hence advancing the development of agentic LLMs capable of robust tool use.
Paper Structure (38 sections, 4 figures, 1 table)

This paper contains 38 sections, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Motivation
  3. Challenges in Data Acquisition
  4. Requirements for Synthetic Agentic Data
  5. Research Questions
  6. Related Work
  7. Web-Based and Interactive Agent Environments
  8. Benchmarks for Tool-Augmented Agents
  9. Agentic Dataset Construction and Synthetic Data Generation
  10. Multi-Agent Frameworks and Conversational Synthesis
  11. Tool Use, Function Calling, and Execution Traces
  12. Data Quality, Risks, and Governance
  13. Methodology
  14. RecordSynth: Multitask Agentic Data from Scratch
  15. Inputs and Outputs
  16. ...and 23 more sections

Figures (4)

  • Figure 1: End-to-end pipeline for generating multitask agentic data, from enterprise-level scenario construction to executable agentic flows with ground-truth functions.
  • Figure 2: Agentic Data with DAG First Approach
  • Figure 4: Turn-level interaction flow in MultiTurnDialogueSynth: each step consists of user input generation, agent tool invocation, argument validation, mocked response generation, and judge feedback.
  • Figure 5: Agentic Record Rollout Pipeline. Inputs (agentic records, prompt templates, tokenizer) flow through validation, instantiation, conversation assembly, and tokenization/bucketing to produce serialized JSON datasets and rollout statistics.