ECR: Manifold-Guided Semantic Cues for Compact Language Models

TL;DR

ECR tackles the degradation of embedding-space geometry in compact multilingual models by introducing embedding consistency regulation. It uses teacher-derived semantic anchors to project and discretize embeddings into input-prefix control tokens, preserving manifold structure without altering the decoder or requiring extra teacher passes. By evaluating on English, Chinese, and Hindi with 1B and 3B models, ECR improves stability, cross-lingual coherence, and downstream task alignment, often allowing smaller models to outperform larger baselines in semantic metrics. The method demonstrates strong potential for on-device deployment, privacy-preserving settings, and compatibility with existing distillation pipelines while offering a principled geometry-based alternative to standard KD. Overall, ECR shows that maintaining representation geometry can be more crucial than pointwise logit matching for compact multilingual models.

Abstract

Compact models often lose the structure of their embedding space. The issue shows up when the capacity is tight or the data spans several languages. Such collapse makes it difficult for downstream tasks to build on the resulting representation. Existing compression methods focus on aligning model outputs at a superficial level but fail to preserve the underlying manifold structure. This mismatch often leads to semantic drift in the compact model, causing both task behavior and linguistic properties to deviate from the reference model. To address those issues, we provide a new framework called Embedding Consistency Regulation (ECR). This framework first derives a set of semantic anchors from teacher embeddings (computed once offline). Then, the compact model learns to maintain consistent geometry around these anchors, without relying on matching logits or internal features. ECR adds only a small projection step at inference, without altering the decoding architecture or its runtime behavior. In experiments on a 100K multilingual corpus, ECR consistently stabilizes training and preserves semantic structure across tasks and languages. It also produces a more compact and task-aligned representation space, enabling low-capacity models to learn cleaner manifolds than conventional baselines. ECR works without teacher outputs and is compatible with, but independent of, distillation. Taken together, our results show that ECR helps compact models better follow task requirements and makes them easier to deploy under strict efficiency or privacy limits.

Figures (6)

• Figure 1: Illustration of Embedding Consistency Regulation (ECR). A query embedding is projected onto semantic manifold anchors through $\mathcal{P}_i$, yielding a discretized control signal $c_i$. These control codes are converted into prefix control tokens and inserted at the input layer, providing geometric cues that encourage the compact model to maintain more stable semantic organization during training. ECR operates entirely through input conditioning---no auxiliary losses, feature matching, or architectural modification are required. The symbolic gradient notation is used only to illustrate the indirect effect that changed inputs have on the optimization trajectory; ECR does not modify gradients explicitly.
• Figure 2: Semantic factorization and control-token construction in ECR. Each input query (left) is projected onto a structured semantic manifold consisting of primary factors (task, language) and multiple auxiliary dimensions (emotion, intent, tone, strategy). The projection identifies the closest semantic anchors, which are discretized into factor-specific control codes (T, L, E, I, P). These codes are emitted as prefix control tokens (right), forming an interpretable, factor-aligned conditioning signal that guides the compact model during generation. This figure illustrates two examples (Chinese and Hindi queries), showing that ECR handles multilingual and multi-factor semantics through the same projection--prefix pipeline.
• Figure 3: Training loss curves for all configurations. FP32 models converge normally at the beginning, but without ECR the baseline cannot remain stable and its loss eventually explodes under the no–gradient-clipping setting. In contrast, the 1B FP32+ECR model stays stable throughout. The 3B BF16 model also converges, but its multi-task training signal makes it less steady than the 1B FP32+ECR run. Dashed lines mark epoch boundaries.
• Figure 4: Ablation loss curves (overview). All manifold subsets exhibit nearly identical optimization behavior during the first epoch, indicating that ECR does not destabilize training even under aggressive regularization settings. The highlighted point corresponds to the region magnified in Fig. \ref{['fig:ablation_zoom']}.
• Figure 5: Ultra–zoom-in view of the ablation loss curves around step 644–645. All curves remain nearly perfectly parallel, confirming that geometric regularization affects manifold structure without perturbing the local optimization dynamics.