QAQ: Bidirectional Semantic Coherence for Selecting High-Quality Synthetic Code Instructions

Abstract

Synthetic data has become essential for training code generation models, yet it introduces significant noise and hallucinations that are difficult to detect with current metrics. Existing data selection methods like Instruction-Following Difficulty (IFD) typically assess how hard a model generates an answer given a query ($A|Q$). However, this metric is ambiguous on noisy synthetic data, where low probability can distinguish between intrinsic task complexity and model-generated hallucinations. Here, we propose QAQ, a novel data selection framework that evaluates data quality from the reverse direction: how well can the answer predict the query ($Q|A$)? We define Reverse Mutual Information (RMI) to quantify the information gain about the query conditioned on the answer. Our analyses reveal that both extremes of RMI signal quality issues: low RMI indicates semantic misalignment, while excessively high RMI may contain defect patterns that LLMs easily recognize. Furthermore, we introduce a selection strategy based on the disagreement between strong and weak models to identify samples that are valid yet challenging. Experiments on the WarriorCoder dataset demonstrate that selecting just 25% of data using stratified RMI achieves comparable performance to full-data training, significantly outperforming existing data selection methods. Our approach highlights the importance of bidirectional semantic coherence in synthetic data curation, offering a scalable pathway to reduce computational costs without sacrificing model capability.

Figures (5)

• Figure 1: (a) Two common failure modes in synthetic $(Q,A)$: keyword echo/paraphrase (high RMI) and misalignment (low RMI). (b) The comparisons between our method and traditional IFD method. Unlike previous work, QAQ scores question predictability given the answer $(Q | A)$ and computes RMI. We keep the mid-range "sweet spot" and filter both extremes.
• Figure 2: Overall workflow of the QAQ framework for synthetic code data selection. (Top-left) RMI is calculated as the information gain provided by the answer about the query, using reverse perplexity. (Bottom-left) Our analysis identifies that RMI extremes signal quality issues: low RMI indicates semantic misalignment, while excessively high RMI indicates trivial paraphrasing or echoing. (Center) To eliminate the confounding factor of question complexity, RMI is stratified into 10 bins based on $\text{PPL}(Q)$ to compute normalized within-bin ranks $r$. (Right) The final selection targets the "Cognitive Gap" ($\text{Diff} = r_s - r_w$) between a strong model $M_s$ and a weak model $M_w$. Samples that are recognized as high-quality only by the strong model are retained, resulting in a high-quality 25% subset $S$ that provides the strongest learning signal.
• Figure 3: Scatter plot of RMI (Question-based) vs. -log(IFD) (Answer-based). Despite both being mutual information estimates, the Spearman rank correlation is low ($\rho = 0.252$), suggesting they capture different aspects of data quality.
• Figure 4: Scatter plot of RMI vs. $\log \text{PPL}(Q)$. The heteroscedastic pattern (low variance at low PPL, high variance at high PPL) motivates stratification by question complexity.
• Figure 5: Scatter plot comparing Standard RMI vs. Raw RMI. The lack of system context in the Raw version leads to inflated $PPL(Q)$ and subsequent overestimation of RMI.