Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Capability Ceilings in Autoregressive Language Models: Empirical Evidence from Knowledge-Intensive Tasks

Javier Marín

TL;DR

This work evaluates whether simply increasing parameter counts in decoder-only autoregressive transformers improves knowledge-intensive versus procedural task performance. Using OPT and Pythia models from 70M to 30B parameters, it reports accuracy and cross-entropy loss across knowledge tasks like MMLU mathematics, arithmetic, and QQP, plus attention-intervention tests. Key findings show knowledge-task accuracy remains flat around $19$–$20$ percent despite a substantial loss reduction of about $31$ percent, while arithmetic scales normally from about $2.4$ percent to $31$ percent, and attention-pattern swaps can cause catastrophic degradation, indicating brittle, pattern-driven representations. These results warn against relying on parameter scaling alone for knowledge tasks in decoder-only architectures and motivate exploring retrieval-augmented or hybrid architectures, though generalization to other designs remains an open question.

Abstract

We document empirical capability ceilings in decoder-only autoregressive language models across knowledge-intensive tasks. Systematic evaluation of OPT and Pythia model families (70M-30B parameters, spanning 240 times scaling) reveals that knowledge retrieval tasks show negligible accuracy improvement despite smooth loss reduction. On MMLU mathematics benchmarks, accuracy remains flat at 19-20% (below 25% random chance) across all scales while cross-entropy loss decreases by 31%. In contrast, procedural tasks like arithmetic show conventional scaling where both metrics improve together. Attention intervention experiments reveal high sensitivity to perturbation: swapping attention patterns between models causes catastrophic performance collapse (complete accuracy loss) rather than graceful degradation. These measurements have immediate engineering implications: for knowledge-intensive applications using OPT and Pythia architectures, parameter scaling beyond 1-2B offers minimal accuracy gains despite continued loss improvement. Our findings quantify capability-specific scaling failures in these model families to inform resource allocation decisions. Whether these patterns reflect fundamental constraints of decoder-only architectures or implementation-specific limitations remains an open question requiring investigation across diverse architectural approaches.

Capability Ceilings in Autoregressive Language Models: Empirical Evidence from Knowledge-Intensive Tasks

TL;DR

This work evaluates whether simply increasing parameter counts in decoder-only autoregressive transformers improves knowledge-intensive versus procedural task performance. Using OPT and Pythia models from 70M to 30B parameters, it reports accuracy and cross-entropy loss across knowledge tasks like MMLU mathematics, arithmetic, and QQP, plus attention-intervention tests. Key findings show knowledge-task accuracy remains flat around – percent despite a substantial loss reduction of about percent, while arithmetic scales normally from about percent to percent, and attention-pattern swaps can cause catastrophic degradation, indicating brittle, pattern-driven representations. These results warn against relying on parameter scaling alone for knowledge tasks in decoder-only architectures and motivate exploring retrieval-augmented or hybrid architectures, though generalization to other designs remains an open question.

Abstract

We document empirical capability ceilings in decoder-only autoregressive language models across knowledge-intensive tasks. Systematic evaluation of OPT and Pythia model families (70M-30B parameters, spanning 240 times scaling) reveals that knowledge retrieval tasks show negligible accuracy improvement despite smooth loss reduction. On MMLU mathematics benchmarks, accuracy remains flat at 19-20% (below 25% random chance) across all scales while cross-entropy loss decreases by 31%. In contrast, procedural tasks like arithmetic show conventional scaling where both metrics improve together. Attention intervention experiments reveal high sensitivity to perturbation: swapping attention patterns between models causes catastrophic performance collapse (complete accuracy loss) rather than graceful degradation. These measurements have immediate engineering implications: for knowledge-intensive applications using OPT and Pythia architectures, parameter scaling beyond 1-2B offers minimal accuracy gains despite continued loss improvement. Our findings quantify capability-specific scaling failures in these model families to inform resource allocation decisions. Whether these patterns reflect fundamental constraints of decoder-only architectures or implementation-specific limitations remains an open question requiring investigation across diverse architectural approaches.
Paper Structure (22 sections, 1 equation, 3 figures, 2 tables)

This paper contains 22 sections, 1 equation, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Empirical Contributions
  3. Scope and Limitations
  4. Practical Implications
  5. Related Work
  6. Neural Scaling Laws
  7. Emergent Capabilities and Their Critiques
  8. Knowledge in Language Models
  9. Architectural Alternatives
  10. Experimental Setup
  11. Model Selection
  12. Task Selection
  13. Evaluation Protocol
  14. Attention Intervention Experiments
  15. Results
  16. ...and 7 more sections

Figures (3)

  • Figure 1: Accuracy scaling patterns across task types. MMLU shows flat accuracy below random chance (25%) across all scales, while arithmetic demonstrates conventional scaling. QQP (paraphrase detection) shows minimal scaling with accuracy fluctuating around 60%. Pattern is consistent across both OPT and Pythia families.
  • Figure 2: Cross-entropy loss decreases smoothly across all tasks and both model families, regardless of whether accuracy improves. This demonstrates that loss alone masks capability stagnation.
  • Figure 3: Normalized performance metrics show MMLU loss and accuracy completely decouple as models scale, while arithmetic metrics improve together. Models learn to confidently produce wrong answers on knowledge tasks.