DUET: Optimizing Training Data Mixtures via Feedback from Unseen Evaluation Tasks

TL;DR

DUET addresses unseen evaluation tasks where the deployed data are unknown by optimizing training data mixtures through a bilevel, global-to-local approach. The outer level uses Bayesian optimization to adjust domain mixing ratios $\mathbf{r}$ on the simplex, while the inner level uses an influence-function (IF) based estimator to greedily assemble high-quality data under the proposed mix, yielding $\widetilde{y_r^*}$. The authors prove a sublinear cumulative regret bound for DUET and demonstrate its effectiveness across in-domain and out-of-domain language tasks, outperforming existing data-mixing and data-selection baselines. The work highlights practical gains for LLM fine-tuning with limited, feedback-driven rounds and suggests extensions to pre-training and privacy-preserving settings. The approach combines principled BO with data-point influence signals to efficiently tailor data mixtures to unseen evaluation objectives, providing a new tool for task-specific data curation under privacy constraints.

Abstract

The performance of an LLM depends heavily on the relevance of its training data to the downstream evaluation task. However, in practice, the data involved in an unseen evaluation task is often unknown (e.g., conversations between an LLM and a user are end-to-end encrypted). Hence, it is unclear what data are relevant for fine-tuning the LLM to maximize its performance on the specific unseen evaluation task. Instead, one can only deploy the LLM on the unseen task to gather multiple rounds of feedback on how well the model performs (e.g., user ratings). This novel setting offers a refreshing perspective towards optimizing training data mixtures via feedback from an unseen evaluation task, which prior data mixing and selection works do not consider. Our paper presents DUET, a novel global-to-local algorithm that interleaves influence function as a data selection method with Bayesian optimization to optimize data mixture via feedback from a specific unseen evaluation task. By analyzing DUET's cumulative regret, we theoretically show that DUET converges to the optimal training data mixture for an unseen task even without any data knowledge of the task. Finally, our experiments across a variety of language tasks demonstrate that DUET outperforms existing data selection and mixing methods in the unseen-task setting.

Figures (6)

• Figure 1: DUET exploits a feedback loop to optimize the data mixture for an unseen evaluation task.
• Figure 2: Empirical distribution of the uniform random and IF-driven estimator $\widetilde{y_r^*}$. Red line is the true inner problem solution.
• Figure 3: Results on unseen LLM evaluation task domains over 10 iterations (higher is better) for Llama-3-8b-Instruct. Experiments were repeated with Qwen2.5-7b-Instruct in App. \ref{['app:addition-results-qwen']}. The subcaption indicates the evaluation task. Underlined evaluation tasks are more difficult because the evaluation task domains are removed from the training data (i.e., out-of-domain).
• Figure 4: Ablation of different components of DUET.
• Figure 7: (a): Empirical distribution of evaluation task accuracy $\mathcal{L}_{\textrm{eval}}(\theta_{\mathcal{X}})$ from each data mixture sample $\mathcal{X}$ (b): empirical distribution of the estimators introduced in Sec. \ref{['sec: solve inner']}. The green histogram is our method of performing IF-weighted sampling to obtain data mixtures. The gray histogram is simply randomly sampling data mixtures with no data selection methods. The purple histogram is the method of removing 20% of the data points with the lowest IF values.

Theorems & Definitions (13)

• Theorem 3.1
• Theorem 3.2
• Theorem 4.1
• Theorem C.1
• proof
• Lemma C.0
• Theorem C.1
• proof
• Theorem C.1
• proof