Recursive KL Divergence Optimization: A Dynamic Framework for Representation Learning

TL;DR

RKDO reframes representation learning as a dynamic, recursive alignment of neighborhood conditional distributions, extending static KL objectives like I-Con to the entire response field. It provides a formal recurrence for supervisor and model distributions and proves linear-rate convergence, supported by experiments on CIFAR-10/100 and STL-10 that show about 30% lower training losses and 60–80% reductions in training updates. The work analyzes the trade-offs between optimization efficiency and generalization, offering design guidelines for recursion depth and parameter schedules, and demonstrates substantial practical gains in resource-constrained settings. Overall, RKDO introduces a principled, recursive framework that accelerates early learning and opens avenues for applying divergence-based objectives to dynamic, structured neighborhoods across tasks and modalities.

Abstract

We propose a generalization of modern representation learning objectives by reframing them as recursive divergence alignment processes over localized conditional distributions While recent frameworks like Information Contrastive Learning I-Con unify multiple learning paradigms through KL divergence between fixed neighborhood conditionals we argue this view underplays a crucial recursive structure inherent in the learning process. We introduce Recursive KL Divergence Optimization RKDO a dynamic formalism where representation learning is framed as the evolution of KL divergences across data neighborhoods. This formulation captures contrastive clustering and dimensionality reduction methods as static slices while offering a new path to model stability and local adaptation. Our experiments demonstrate that RKDO offers dual efficiency advantages approximately 30 percent lower loss values compared to static approaches across three different datasets and 60 to 80 percent reduction in computational resources needed to achieve comparable results. This suggests that RKDOs recursive updating mechanism provides a fundamentally more efficient optimization landscape for representation learning with significant implications for resource constrained applications.

Figures (5)

• Figure 1: RKDO Loss Improvement Over I-Con: Consistent percentage improvement in loss values achieved by RKDO compared to I-Con across all datasets and training durations. Note that the figure labels use RKDO to refer to what is described as RKDO in this paper.
• Figure 2: RKDO Loss Improvement Over I-Con: Consistent percentage improvement in loss values achieved by RKDO compared to I-Con across all datasets and training durations.
• Figure 3: Computational Efficiency: RKDO at 2 epochs achieves comparable or superior performance to I-Con at 5 epochs across datasets, demonstrating significant computational resource savings.
• Figure 4: Early Learning: RKDO vs. I-Con at 1 and 2 epochs shows RKDO's advantage at 2 epochs on CIFAR-100 and STL-10, with comparable performance on CIFAR-10.
• Figure 5: The figure shows a line graph titled "RKDO vs. ICon: Loss Across Training Durations" comparing the performance of two representation learning approaches (RKDO and ICon) across different datasets (CIFAR-10, CIFAR-100, and STL-10) and training durations (1, 2, 5, and 10 epochs). The graph clearly illustrates that RKDO (solid lines) consistently achieves lower loss values than ICon (dashed lines) across all datasets and training durations, demonstrating RKDO's superior optimization efficiency.

Theorems & Definitions (6)

• Lemma 1: Jensen Step
• proof
• Lemma 2: Inner Minimization
• proof
• Theorem 3: Linear-Rate Convergence
• proof