The Prompt is Mightier than the Example

TL;DR

The paper addresses the challenge of high ICL costs in synthetic tabular data generation with LLMs by introducing Knowledge-Guided Prompting (KGP), which injects global domain knowledge into prompts to complement or substitute for in-context examples. It formalizes three knowledge types—Symbolic, Semantic, and Statistical—and demonstrates a data-driven scaling law that relates knowledge level and ICL count to generation quality. Through extensive experiments on math, geometry, and real-world datasets, the authors show that KGP can reduce ICL requirements by 40–90% (depending on data complexity) while improving low-order statistics, distributional fidelity, and ML utility, and they validate these gains in a noisy cyber-physical case study. The findings indicate that KGP enables scalable, robust synthetic data generation with practical impact for privacy-preserving ML, data augmentation, and out-of-distribution generalization, with future work extending to multimodal knowledge integration and defense against potentially inconsistent guidance.

Abstract

Numerous recent prompt optimization approaches like chain-of-thought, have been demonstrated to significantly improve the quality of content generated by large language models (LLMs). In-context learning (ICL), a recent paradigm where a few representative examples guide content generation has also led to strong improvements in generation quality of LLM generated content. This idea has been applied to great effect in synthetic tabular data generation, where LLMs, through effective use of ICL and prompt optimization, can generate data that approximate samples from complex, heterogeneous distributions based on representative examples. However, ensuring high-fidelity synthetic data often requires a very large number of ICL examples which may be unavailable or costly to obtain. At the same time, as LLMs get larger and larger, their in-built prior knowledge becomes vast and can potentially substitute for specific data examples. In this paper, we introduce Knowledge-Guided Prompting (KGP) as a new knob in prompt optimization and explore the ability of KGP-based prompt optimization to offset the cost of ICL. Specifically, we explore the question `how many examples can a prompt substitute for?' and explore knowledge-guided prompting (KGP) where domain knowledge, either inferred or available, is explicitly injected into the prompt, reducing dependence on ICL examples. Our experiments systematically explore the trade-off between ICL and KGP, revealing an empirical scaling law that quantifies how quality of generated synthetic data varies with increasing domain knowledge and decreasing example count. Our results demonstrate that knowledge-guided prompting can be a scalable alternative, or addition, to in-context examples, unlocking new approaches to synthetic data generation.

Figures (5)

• Figure 1: (a) a traditional synthetic tabular data generation pipeline using LLMs encodes sample data as in-context learning examples to drive the generation process. (b) Our prompt optimization approach based on knowledge-guided prompting (KGP), incorporates automatically inferred domain knowledge, providing the LLM-based generator a complementary context in addition to ICL examples. Our experimental findings indicate that such global property conditioning via. KGP leads to a significant improvement in synthetic data generation quality, indicating that KGP can indeed be employed as a useful knob for prompt optimization.
• Figure 2: Showcasing the MAPE and Hustoff distance between the synthetic data and the real data. X-axis represents different ICL data sizes. The green curve represents the semantic KGP and the blue curve represents the No-KGP setting. Take (a) for example, by incorporating the visual knowledge phrase "x and y coordinates of points when plotted visually depict a dinosaur." into the prompt, the quality of the generated data improves when the dataset is limited. The quantitative metric Hausdorff Distance decreased from 18.54 to 7.72 indicating a significant improvement when using 60 In-Context Samples.
• Figure 3: Visualization of out-of-distribution (OOD) generation, featuring two mathematical functions: Sigmoid and Bohachevsky. In the ICL Real Data figure (a) & (e), the red data points represent the observed field, whereas the grey data points indicate the complete ground truth field. Figure (b)-(d), (f)-(h) showcase the generated synthetic data under corresponding KGP settings.
• Figure 4: Diversity of modes in synthetic data. Five columns from left to the right are real data, No KGP, statistical KGP, semantic KGP, and symbolic KGP.
• Figure 5: Diversity of modes in synthetic data. Three columns from left to the right represent real data, No KGP, Semantic KGP.