Switchable Decision: Dynamic Neural Generation Networks
Shujian Zhang, Korawat Tanwisuth, Chengyue Gong, Pengcheng He, Mingyuan Zhou
TL;DR
This work tackles the sluggish inference of autoregressive generation models by introducing a switchable decision framework that dynamically allocates computation during generation. It defines a discrete decision space over attention, feed-forward, and token candidates and trains an RL-based policy to decide per-input inference paths under a lexicographic constrained optimization that prioritizes quality while reducing FLOPs. The approach is demonstrated across summarization, QA, and classification, achieving up to around 40% FLOPs reduction with negligible accuracy loss and providing extensive ablations to validate design choices. The method is generalizable across architectures and generation tasks, offering a practical plug-in for real-time NLP systems and setting a new direction for efficient, adaptive inference in language generation.
Abstract
Auto-regressive generation models achieve competitive performance across many different NLP tasks such as summarization, question answering, and classifications. However, they are also known for being slow in inference, which makes them challenging to deploy in real-time applications. We propose a switchable decision to accelerate inference by dynamically assigning computation resources for each data instance. Automatically making decisions on where to skip and how to balance quality and computation cost with constrained optimization, our dynamic neural generation networks enforce the efficient inference path and determine the optimized trade-off. Experiments across question answering, summarization, and classification benchmarks show that our method benefits from less computation cost during inference while keeping the same accuracy. Extensive experiments and ablation studies demonstrate that our method can be general, effective, and beneficial for many NLP tasks.