Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mixing Configurations for Downstream Prediction

Juntang Wang, Hao Wu, Runkun Guo, Yihan Wang, Dongmian Zou, Shixin Xu

TL;DR

The paper addresses learning from multi-resolution clustering configurations to improve downstream prediction in settings with limited labels. It proposes GraMixC, a plug-and-play module that extracts configurations, aligns them with RMS, and fuses them through attention-based mechanisms for task- and input-specific mixing. Key contributions include identifying three properties of configurations, achieving state-of-the-art results on a challenging DSNI 16S rRNA cultivation-media task, and validating gains across standard tabular benchmarks. The work demonstrates that unsupervised, multi-resolution priors can be effectively leveraged to boost predictive performance with a scalable fusion strategy.

Abstract

Humans possess an innate ability to group objects by similarity, a cognitive mechanism that clustering algorithms aim to emulate. Recent advances in community detection have enabled the discovery of configurations -- valid hierarchical clusterings across multiple resolution scales -- without requiring labeled data. In this paper, we formally characterize these configurations and identify similar emergent structures in register tokens within Vision Transformers. Unlike register tokens, configurations exhibit lower redundancy and eliminate the need for ad hoc selection. They can be learned through unsupervised or self-supervised methods, yet their selection or composition remains specific to the downstream task and input. Building on these insights, we introduce GraMixC, a plug-and-play module that extracts configurations, aligns them using our Reverse Merge/Split (RMS) technique, and fuses them via attention heads before forwarding them to any downstream predictor. On the DSN1 16S rRNA cultivation-media prediction task, GraMixC improves the R2 score from 0.6 to 0.9 across multiple methods, setting a new state of the art. We further validate GraMixC on standard tabular benchmarks, where it consistently outperforms single-resolution and static-feature baselines.

Mixing Configurations for Downstream Prediction

TL;DR

The paper addresses learning from multi-resolution clustering configurations to improve downstream prediction in settings with limited labels. It proposes GraMixC, a plug-and-play module that extracts configurations, aligns them with RMS, and fuses them through attention-based mechanisms for task- and input-specific mixing. Key contributions include identifying three properties of configurations, achieving state-of-the-art results on a challenging DSNI 16S rRNA cultivation-media task, and validating gains across standard tabular benchmarks. The work demonstrates that unsupervised, multi-resolution priors can be effectively leveraged to boost predictive performance with a scalable fusion strategy.

Abstract

Humans possess an innate ability to group objects by similarity, a cognitive mechanism that clustering algorithms aim to emulate. Recent advances in community detection have enabled the discovery of configurations -- valid hierarchical clusterings across multiple resolution scales -- without requiring labeled data. In this paper, we formally characterize these configurations and identify similar emergent structures in register tokens within Vision Transformers. Unlike register tokens, configurations exhibit lower redundancy and eliminate the need for ad hoc selection. They can be learned through unsupervised or self-supervised methods, yet their selection or composition remains specific to the downstream task and input. Building on these insights, we introduce GraMixC, a plug-and-play module that extracts configurations, aligns them using our Reverse Merge/Split (RMS) technique, and fuses them via attention heads before forwarding them to any downstream predictor. On the DSN1 16S rRNA cultivation-media prediction task, GraMixC improves the R2 score from 0.6 to 0.9 across multiple methods, setting a new state of the art. We further validate GraMixC on standard tabular benchmarks, where it consistently outperforms single-resolution and static-feature baselines.
Paper Structure (17 sections, 5 equations, 13 figures, 2 tables, 1 algorithm)

This paper contains 17 sections, 5 equations, 13 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminary results
  3. Methodology
  4. Multi-resolution graph-based clustering
  5. RMS: reverse merge & split alignment
  6. Experiments
  7. Implementation details and experimental setup
  8. Evaluation of the proposed module
  9. Qualitative evaluation of configurations.
  10. Conclusion
  11. An Intuitive Example of Configuration Mixing
  12. DSNI dataset distribution
  13. Synthetic Clustering Benchmarks
  14. RMS Alignment Details
  15. Additional Experimental Results
  16. ...and 2 more sections

Figures (13)

  • Figure 1: Illustration of CIFAR10 configurations. Each column represents a configuration—clustering at a specific resolution.
  • Figure 2: Comparison of attention maps obtained from configurations and registers, rows for samples. (a): Lineage diagram for configurations, near GT balls are marked yellow. (b): Attention map of configuration tokens in an attention-based linear probing. (c): Attention map of DINOv2-reg register tokens, mean of all patch norms is used. (d): Attention maps over the register tokens, as images.
  • Figure 3: Illustration of another two properties of configurations, grouped by left two and right two. (a): Lineage diagram where coarser classes are used for GT. (b): Attention map in linear probing the coarser classes. (c): Distribution of feature vector importance over the register tokens querying, mean of all patch importance is used. (d): Distribution of cosine similarity between query embeddings of register and configuration tokens and their 2 neighbors, mean of all patch similarities is used.
  • Figure 4: Illustration of the proposed GraMixC module and resulting model. The input data branches into (upper) a path to unsupervised learning box that extracts configurations, and (lower) a direct path to the downstream predictor. Their outcomes concatenate and pass to the downstream predictor. The components occur only during training and inference are colored in blue and gray, respectively.
  • Figure 5: Example of the RMS alignment process applied to clustering results and ground truth (both treated as configurations) on the Salinas dataset plazaAutomatedSelectionResults2005. (a): Bipartite graph representation, where blue nodes correspond to predicted clusters and red nodes to ground truth clusters. Node labels indicate cluster indices; edge labels show the proportion of samples shared between clusters. (b): Proportional confusion matrix ${\bm{C}}$ comparing predicted clusters (horizontal axis) to ground truth clusters (vertical axis). (c): Confusion matrix ${\bm{C}}_\mathrm{tw}$ reordered via the two-walk Laplacian. Notable splits, such as ground truth cluster 8 being divided into clusters 8 and 15 in the prediction, can be resolved through the reverse merge/split procedure.
  • ...and 8 more figures