Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fixed Anchors Are Not Enough: Dynamic Retrieval and Persistent Homology for Dataset Distillation

Muquan Li, Hang Gou, Yingyi Ma, Rongzheng Wang, Ke Qin, Tao He

TL;DR

RETA -- a Retrieval and Topology Alignment framework for decoupled DD consistently outperforms various baselines under comparable time and memory, especially reaching 64.3% top-1 accuracy on ImageNet-1K with ResNet-18 at 50 images per class.

Abstract

Decoupled dataset distillation (DD) compresses large corpora into a few synthetic images by matching a frozen teacher's statistics. However, current residual-matching pipelines rely on static real patches, creating a fit-complexity gap and a pull-to-anchor effect that reduce intra-class diversity and hurt generalization. To address these issues, we introduce RETA -- a Retrieval and Topology Alignment framework for decoupled DD. First, Dynamic Retrieval Connection (DRC) selects a real patch from a prebuilt pool by minimizing a fit-complexity score in teacher feature space; the chosen patch is injected via a residual connection to tighten feature fit while controlling injected complexity. Second, Persistent Topology Alignment (PTA) regularizes synthesis with persistent homology: we build a mutual k-NN feature graph, compute persistence images of components and loops, and penalize topology discrepancies between real and synthetic sets, mitigating pull-to-anchor effect. Across CIFAR-100, Tiny-ImageNet, ImageNet-1K, and multiple ImageNet subsets, RETA consistently outperforms various baselines under comparable time and memory, especially reaching 64.3% top-1 accuracy on ImageNet-1K with ResNet-18 at 50 images per class, +3.1% over the best prior.

Fixed Anchors Are Not Enough: Dynamic Retrieval and Persistent Homology for Dataset Distillation

TL;DR

RETA -- a Retrieval and Topology Alignment framework for decoupled DD consistently outperforms various baselines under comparable time and memory, especially reaching 64.3% top-1 accuracy on ImageNet-1K with ResNet-18 at 50 images per class.

Abstract

Decoupled dataset distillation (DD) compresses large corpora into a few synthetic images by matching a frozen teacher's statistics. However, current residual-matching pipelines rely on static real patches, creating a fit-complexity gap and a pull-to-anchor effect that reduce intra-class diversity and hurt generalization. To address these issues, we introduce RETA -- a Retrieval and Topology Alignment framework for decoupled DD. First, Dynamic Retrieval Connection (DRC) selects a real patch from a prebuilt pool by minimizing a fit-complexity score in teacher feature space; the chosen patch is injected via a residual connection to tighten feature fit while controlling injected complexity. Second, Persistent Topology Alignment (PTA) regularizes synthesis with persistent homology: we build a mutual k-NN feature graph, compute persistence images of components and loops, and penalize topology discrepancies between real and synthetic sets, mitigating pull-to-anchor effect. Across CIFAR-100, Tiny-ImageNet, ImageNet-1K, and multiple ImageNet subsets, RETA consistently outperforms various baselines under comparable time and memory, especially reaching 64.3% top-1 accuracy on ImageNet-1K with ResNet-18 at 50 images per class, +3.1% over the best prior.
Paper Structure (15 sections, 1 theorem, 11 equations, 6 figures, 4 tables)

This paper contains 15 sections, 1 theorem, 11 equations, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Preliminary
  4. Method
  5. Dynamic Retrieval Connection
  6. Formulation.
  7. Persistent Topology Alignment
  8. Experiment
  9. Experimental Setup
  10. Comparison with the State-of-the-art
  11. Ablation Study
  12. Computational Efficiency
  13. Robustness to Corruptions
  14. Practical Application
  15. Conclusion

Key Result

Theorem 4.1

Let $\mathfrak{R}_n(\cdot)$ denote the empirical Rademacher complexity and let $H\circ\mathcal{S}$ be the composition of a hypothesis class $H$ with a sample set $\mathcal{S}$. Consider a pre-connection synthetic set $\tilde{\mathcal{C}}_{\mathrm{pre}}$ and a post-connection set $\tilde{\mathcal{C}} where $L\!>\!0$ is a constant depending on $H$ and the connection operator, and $\Delta \!=\!\frac{

Figures (6)

  • Figure 1: Comparison between existing pipelines and our proposed RETA. (a) Decoupled methods squeeze the original dataset into a compact set. (b) Previous residual matching methods FADRM connect a fixed real patch to the distilled images at every step. (c) RETA augments residual matching with Dynamic Retrieval Connection (DRC) and Persistent Topology Alignment (PTA).
  • Figure 2: Overview of RETA. Dynamic Retrieval Connection (top) computes a fit–complexity score to adaptively retrieve real patches as anchors for residual matching, while Persistent Topology Alignment (bottom) aligns the persistent diagrams of real and synthetic features. The two modules are jointly optimized to produce synthetic datasets that retain both accuracy and topology-aware structure.
  • Figure 3: Visualization of images distilled by FADRM and RETA. The synthetic images on Tiny-ImageNet, ImageNette and ImageNet-$1$K are demonstrated here.
  • Figure 4: Ablation studies for hyperparameters on ImageNet-$1$K. Curves show the mean test accuracy, and the error bars denote one standard deviation over five independent runs.
  • Figure 5: Computational efficiency comparison between baseline methods and RETA when distilling ImageNet-$1$k. The time cost is measured in seconds, representing the duration required to generate a single image on a single RTX $4090$ GPU.
  • ...and 1 more figures

Theorems & Definitions (1)

  • Theorem 4.1