Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CTPD: Cross Tokenizer Preference Distillation

Truong Nguyen, Phi Van Dat, Ngan Nguyen, Linh Ngo Van, Trung Le, Thanh Hong Nguyen

TL;DR

CTPD tackles the practical challenge of distilling human preferences from a high-capacity teacher to a smaller student when their tokenizers differ. It combines Aligned Span Projection, a cross-tokenizer extension of Token-level Importance Sampling DPO, and a Teacher-Anchored Reference to enable fine-grained, white-box transfer of preference signals across heterogeneous tokenizers, with theoretical grounding in importance sampling. Empirically, CTPD delivers consistent gains over strong preference-alignment and KD baselines across diverse benchmarks, including GSM8k and TruthfulQA. By decoupling supervision from tokenizer compatibility, CTPD broadens access to efficient, robust alignment and opens avenues for broader cross-tokenizer knowledge transfer.

Abstract

While knowledge distillation has seen widespread use in pre-training and instruction tuning, its application to aligning language models with human preferences remains underexplored, particularly in the more realistic cross-tokenizer setting. The incompatibility of tokenization schemes between teacher and student models has largely prevented fine-grained, white-box distillation of preference information. To address this gap, we propose Cross-Tokenizer Preference Distillation (CTPD), the first unified framework for transferring human-aligned behavior between models with heterogeneous tokenizers. CTPD introduces three key innovations: (1) Aligned Span Projection, which maps teacher and student tokens to shared character-level spans for precise supervision transfer; (2) a cross-tokenizer adaptation of Token-level Importance Sampling (TIS-DPO) for improved credit assignment; and (3) a Teacher-Anchored Reference, allowing the student to directly leverage the teacher's preferences in a DPO-style objective. Our theoretical analysis grounds CTPD in importance sampling, and experiments across multiple benchmarks confirm its effectiveness, with significant performance gains over existing methods. These results establish CTPD as a practical and general solution for preference distillation across diverse tokenization schemes, opening the door to more accessible and efficient alignment of language models.

CTPD: Cross Tokenizer Preference Distillation

TL;DR

CTPD tackles the practical challenge of distilling human preferences from a high-capacity teacher to a smaller student when their tokenizers differ. It combines Aligned Span Projection, a cross-tokenizer extension of Token-level Importance Sampling DPO, and a Teacher-Anchored Reference to enable fine-grained, white-box transfer of preference signals across heterogeneous tokenizers, with theoretical grounding in importance sampling. Empirically, CTPD delivers consistent gains over strong preference-alignment and KD baselines across diverse benchmarks, including GSM8k and TruthfulQA. By decoupling supervision from tokenizer compatibility, CTPD broadens access to efficient, robust alignment and opens avenues for broader cross-tokenizer knowledge transfer.

Abstract

While knowledge distillation has seen widespread use in pre-training and instruction tuning, its application to aligning language models with human preferences remains underexplored, particularly in the more realistic cross-tokenizer setting. The incompatibility of tokenization schemes between teacher and student models has largely prevented fine-grained, white-box distillation of preference information. To address this gap, we propose Cross-Tokenizer Preference Distillation (CTPD), the first unified framework for transferring human-aligned behavior between models with heterogeneous tokenizers. CTPD introduces three key innovations: (1) Aligned Span Projection, which maps teacher and student tokens to shared character-level spans for precise supervision transfer; (2) a cross-tokenizer adaptation of Token-level Importance Sampling (TIS-DPO) for improved credit assignment; and (3) a Teacher-Anchored Reference, allowing the student to directly leverage the teacher's preferences in a DPO-style objective. Our theoretical analysis grounds CTPD in importance sampling, and experiments across multiple benchmarks confirm its effectiveness, with significant performance gains over existing methods. These results establish CTPD as a practical and general solution for preference distillation across diverse tokenization schemes, opening the door to more accessible and efficient alignment of language models.
Paper Structure (22 sections, 2 theorems, 17 equations, 1 figure, 3 tables)

This paper contains 22 sections, 2 theorems, 17 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Preference Alignment
  4. Knowledge Distillation
  5. Methodology
  6. Preliminaries
  7. Discussion: Reference Model as a Reweighting Mechanism.
  8. Aligned Span
  9. Cross Tokenizer Preference Distillation Framework
  10. Importance weight estimation
  11. Experiments
  12. Settings
  13. Baselines and LLMs.
  14. Datasets and Evaluation Metrics.
  15. Hyperparameters.
  16. ...and 7 more sections

Key Result

Theorem 1

Let $r_{w,1}, \dots, r_{w,n_w}$ be a set of $n_w$ independent bounded random variables in $[a_w, b_w]$ representing the rewards of the aligned spans in a winning response. Similarly, let $r_{l,1}, \dots, r_{l,n_l}$ be $n_l$ independent bounded random variables in $[a_l, b_l]$ for a losing response. In this expression, $c_{w,i} = b_w - a_w$ and $c_{l,j} = b_l - a_l$ denote the maximum possible cha

Figures (1)

  • Figure 1: An overview of the Cross-Tokenizer Preference Distillation (CTPD) framework. Initially, both a student and a stronger teacher model are supervised fine-tuned (SFT) using instruction-tuning data. The SFT student model is then further trained using preference data, which consists of winning $y_w$ and losing $y_l$ responses, along with pre-compute aligned span weights. The SFT teacher model serves as a reference to calculate the rewards for aligned spans within these responses. These rewards, along with pre-computed span weights $W$, are ultimately used to compute the objective $L_{CTPD}$, effectively guiding the student model to better align with the preferred outputs.

Theorems & Definitions (4)

  • Definition 1
  • Theorem 1: Label noise at span level
  • Definition 2: Span-level optimal dataset
  • Theorem 2