Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Discriminative vs. Generative classifiers: Rethinking MLLMs for Action Understanding

Zhanzhong Pang, Dibyadip Chatterjee, Fadime Sener, Angela Yao

TL;DR

GAD improves both accuracy and efficiency over generative methods, achieving state-of-the-art results on four tasks across five datasets, including an average 2.5% accuracy gain and 3x faster inference on the largest COIN benchmark.

Abstract

Multimodal Large Language Models (MLLMs) have advanced open-world action understanding and can be adapted as generative classifiers for closed-set settings by autoregressively generating action labels as text. However, this approach is inefficient, and shared subwords across action labels introduce semantic overlap, leading to ambiguity in generation. In contrast, discriminative classifiers learn task-specific representations with clear decision boundaries, enabling efficient one-step classification without autoregressive decoding. We first compare generative and discriminative classifiers with MLLMs for closed-set action understanding, revealing the superior accuracy and efficiency of the latter. To bridge the performance gap, we design strategies that elevate generative classifiers toward performance comparable with discriminative ones. Furthermore, we show that generative modeling can complement discriminative classifiers, leading to better performance while preserving efficiency. To this end, we propose Generation-Assisted Discriminative~(GAD) classifier for closed-set action understanding. GAD operates only during fine-tuning, preserving full compatibility with MLLM pretraining. Extensive experiments on temporal action understanding benchmarks demonstrate that GAD improves both accuracy and efficiency over generative methods, achieving state-of-the-art results on four tasks across five datasets, including an average 2.5% accuracy gain and 3x faster inference on our largest COIN benchmark.

On Discriminative vs. Generative classifiers: Rethinking MLLMs for Action Understanding

TL;DR

GAD improves both accuracy and efficiency over generative methods, achieving state-of-the-art results on four tasks across five datasets, including an average 2.5% accuracy gain and 3x faster inference on the largest COIN benchmark.

Abstract

Multimodal Large Language Models (MLLMs) have advanced open-world action understanding and can be adapted as generative classifiers for closed-set settings by autoregressively generating action labels as text. However, this approach is inefficient, and shared subwords across action labels introduce semantic overlap, leading to ambiguity in generation. In contrast, discriminative classifiers learn task-specific representations with clear decision boundaries, enabling efficient one-step classification without autoregressive decoding. We first compare generative and discriminative classifiers with MLLMs for closed-set action understanding, revealing the superior accuracy and efficiency of the latter. To bridge the performance gap, we design strategies that elevate generative classifiers toward performance comparable with discriminative ones. Furthermore, we show that generative modeling can complement discriminative classifiers, leading to better performance while preserving efficiency. To this end, we propose Generation-Assisted Discriminative~(GAD) classifier for closed-set action understanding. GAD operates only during fine-tuning, preserving full compatibility with MLLM pretraining. Extensive experiments on temporal action understanding benchmarks demonstrate that GAD improves both accuracy and efficiency over generative methods, achieving state-of-the-art results on four tasks across five datasets, including an average 2.5% accuracy gain and 3x faster inference on our largest COIN benchmark.
Paper Structure (26 sections, 4 equations, 10 figures, 16 tables)

This paper contains 26 sections, 4 equations, 10 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Method
  4. Preliminaries
  5. Architecture
  6. Model Training & Inference
  7. Experiments
  8. Datasets, Evaluation, and Implementation Details
  9. Main Results
  10. Ablation Studies
  11. Comparison with SOTA.
  12. Qualitative Results.
  13. Generalization Trade-off
  14. Conclusion
  15. Feature Analysis of Generative vs. Discriminative Classifier
  16. ...and 11 more sections

Figures (10)

  • Figure 1: T-SNE plot comparing feature spaces of generative and discriminative classifiers on CrossTask dataset for actions sharing the verb 'add'. Generative feature - mean of output token features; Discriminative feature - the learnable token feature.
  • Figure 2: Comparison between different architectures for downstream video-related recognition tasks: (1) Generative classifier: treating action labels as free text. (2) Discriminative classifier: learning an extra representation for downstream tasks. (3) Generation-assisted discriminative (GAD) classifier: learning an extra representation that is regularized through task-related generation.
  • Figure 2: Generative (Gen) vs. discriminative (Disc) classifier on COIN for step/task and next step prediction.
  • Figure 3: Bridging generative and discriminative classifiers using different tokenization with Llama3.2-1B-Instruct.
  • Figure 4: Tokenization strategies. $\alpha$ and $\beta$ denote random tokens.
  • ...and 5 more figures