TALON: Test-time Adaptive Learning for On-the-Fly Category Discovery

TL;DR

This work proposes a test-time adaptation framework that enables learning through discovery that incorporates two complementary strategies: a semantic-aware prototype update and a stable test-time encoder update that dynamically refines class prototypes to enhance classification.

Abstract

On-the-fly category discovery (OCD) aims to recognize known categories while simultaneously discovering novel ones from an unlabeled online stream, using a model trained only on labeled data. Existing approaches freeze the feature extractor trained offline and employ a hash-based framework that quantizes features into binary codes as class prototypes. However, discovering novel categories with a fixed knowledge base is counterintuitive, as the learning potential of incoming data is entirely neglected. In addition, feature quantization introduces information loss, diminishes representational expressiveness, and amplifies intra-class variance. It often results in category explosion, where a single class is fragmented into multiple pseudo-classes. To overcome these limitations, we propose a test-time adaptation framework that enables learning through discovery. It incorporates two complementary strategies: a semantic-aware prototype update and a stable test-time encoder update. The former dynamically refines class prototypes to enhance classification, whereas the latter integrates new information directly into the parameter space. Together, these components allow the model to continuously expand its knowledge base with newly encountered samples. Furthermore, we introduce a margin-aware logit calibration in the offline stage to enlarge inter-class margins and improve intra-class compactness, thereby reserving embedding space for future class discovery. Experiments on standard OCD benchmarks demonstrate that our method substantially outperforms existing hash-based state-of-the-art approaches, yielding notable improvements in novel-class accuracy and effectively mitigating category explosion. The code is publicly available at \textcolor{blue}{https://github.com/ynanwu/TALON}.

Figures (6)

• Figure 1: At test time, existing methods (left) rely on static inference and fail to adapt to label space shifts, leading to inaccurate discovery and recognition. In contrast, our TALON (right) continually accumulates knowledge from unlabeled test data to overcome this challenge. Moreover, our hash-free framework(right) replaces heuristic hash-based (left) designs, achieving higher representational expressiveness and better discovery stability.
• Figure 2: Overview of the proposed TALON framework. (a) During the offline stage, we introduce margin-aware logit calibration to enlarge inter-class margins and enhance intra-class compactness, reserving embedding space for future category discovery. (b) At test-time, we jointly update the encoder and class prototypes, enabling the model to learn through discovery rather than static inference.
• Figure 3: Angular analysis of the margin-aware logit calibra-tion on the Pets dataset. Left: angles between samples and their class prototypes; Right: angles between different class prototypes.
• Figure 4: Hyperparameter analysis on CUB-200-2011, Stanford Cars, and Oxford Pets datasets. Each column corresponds to one dataset, showing the effects of adaptation batch size, angular margin $m$, and similarity threshold $\tau$ on accuracy (All, Old, New) and the number of newly discovered categories (NDC).
• Figure 5: Hyperparameter analysis on CUB-200-2011, Stanford Cars, and Oxford Pets datasets. Each column corresponds to one dataset, showing the effects of logit scale $s$ and the smoothing constant $\kappa$ on accuracy (All, Old, New) and the number of newly discovered categories (NDC).