Soft Contextualized Encoder For User Defined Text Classification
Charu Maheshwari, Vyas Raina
TL;DR
The paper addresses UDTC where label sets are dynamic and unseen at training time by introducing a Soft Contextualized Encoder (SCE) that treats labels as context tokens and conditions their representations on the input via a soft prompt. The architecture jointly contextualizes input text and label tokens using a permutation-invariant transformer, while leveraging external frozen embeddings for the input to maintain efficiency. Empirical results across seen and unseen benchmarks show SCE achieving state-of-the-art or competitive accuracy with substantially lower training compute compared to large decoder-based finetuning, demonstrating strong cross-domain generalization and practicality for real-world UDTC tasks. The study also quantifies compute trade-offs against LoRA-tuned LLM baselines, highlighting SCE as a compute-efficient alternative for UDTC at around one billion parameters and pointing to future work on scaling and multi-label extensions.
Abstract
User-Defined Text Classification (UDTC) considers the challenge of classifying input text to user-specified, previously unseen classes, a setting that arises frequently in real-world applications such as enterprise analytics, content moderation, and domain-specific information retrieval. We propose a soft-contextualized encoder architecture for UDTC which contextualizes each candidate label with the label set and a static soft prompt representation of the input query. Training on diverse, multi-source datasets enables the model to generalize effectively to zero-shot classification over entirely unseen topic sets drawn from arbitrary domains. We evaluate the proposed architecture both on held-out in-distribution test data and on multiple unseen UDTC benchmarks. Across datasets, the model achieves state-of-the-art performance, consistently outperforming or matching the baselines.