LLM See, LLM Do: Guiding Data Generation to Target Non-Differentiable Objectives

TL;DR

The paper investigates how synthetic data distillation induces passive inheritance of model properties across LLMs and introduces active inheritance to explicitly steer non-differentiable data attributes during distillation. It presents a large profiling toolkit spanning textual characteristics, social bias, calibration, and toxicity, demonstrating that synthetic data can produce significant, sometimes counterintuitive shifts in generation and evaluation preferences. By leveraging best-of-$k$ sampling and multi-teacher ensembles, active inheritance boosts desirable traits like longer, more diverse text while reducing harmful outputs, and it highlights the dominant role of architecture priors in shaping model preferences. The work offers practical guidance for responsible synthetic-data use, including toxicity mitigation and attribute steering, and contributes a scalable framework for monitoring and controlling LLM-influenced data ecosystems. Overall, it advances understanding of how to harness synthetic data responsibly to attain targeted generation profiles and safer AI systems in real-world evaluation pipelines.

Abstract

The widespread adoption of synthetic data raises new questions about how models generating the data can influence other large language models (LLMs) via distilled data. To start, our work exhaustively characterizes the impact of passive inheritance of model properties by systematically studying the consequences of synthetic data integration. We provide one of the most comprehensive studies to-date of how the source of synthetic data shapes models' internal biases, calibration and generations' textual attributes and preferences. We find that models are surprisingly sensitive towards certain attributes even when the synthetic data prompts appear "neutral". which invites the question whether this sensitivity can be exploited for good. Our findings invite the question can we explicitly steer the models towards the properties we want at test time by exploiting the data generation process? This would have historically been considered infeasible due to the cost of collecting data with a specific characteristic or objective in mind. However, improvement in the quality of synthetic data, as well as a shift towards general-purpose models designed to follow a diverse way of instructions, means this question is timely. We propose active inheritance as a term to describe intentionally constraining synthetic data according to a non-differentiable objective. We demonstrate how active inheritance can steer the generation profiles of models towards desirable non-differentiable attributes, e.g. high lexical diversity or low toxicity.

Figures (6)

• Figure 1: Percentage of change in attributes with respect to the base model after synthetic data distillation. Our targeted sampling approach (active inheritance) effectively steers model behaviour to discrete preferences by enhancing desirable attributes (length, diversity) and mitigating negative ones (toxicity) using both the single-source and multi-source sampling strategies.
• Figure 2: Model profile changes after finetuning LLMs on synthetic data.Left: social bias score changes for the BBQ benchmark show a positive decreasing trend for LLaMa2-13B except in the Disability metric. Middle: small changes in Measure of Textual Lexical Diversity (MTLD) and the Readability Index (Rix) are accompanied by an increase of over 100% for the mean number for tokens. Right: toxicity metrics get worse in all cases after finetuning, increasing up to 40%. Overall, we see that models are susceptible to changes of considerable magnitude and that the direction of change is not always intuitive.
• Figure 3: Agreement (i.e. agreement on the best answer when models are shown the same two pairs of candidate answers) between models finetuned on data collected from different LLMs and original LLaMa2-7B, Mixtral-8x7B and human-annotated data. The x-axis displays the student-teacher combinations analysed and is ordered by human agreement. It can be observed that when models are trained with data distilled from other models their inter-model agreement increases.
• Figure 4: Comparison of active inheritance methods (single-source and multi-source sampling) targeting various metrics, where the goals are to increase length and lexical diversity and decrease toxicity. Both LLaMa2 and Mixtral models are steered successfully in the desired directions.
• Figure 5: Analysis of lexical diversity and length gains when filtering is performed on an increasing number of candidate samples per prompt in the multi-source setting considering LLaMa2-7B, where the colored bars indicate the relative gains and the the hatched grey bars indicate the random sampling baseline results (both relative to the base model). We can see that while there does not seem to be any correlation between higher sample availability and bigger gains for length the more samples distilled the bigger was the effect of active inheritance for the lexical diversity attribute.