Dynamic Universal Approximation Theory: The Basic Theory for Transformer-based Large Language Models

TL;DR

This workため builds a Dynamic Universal Approximation Theory (DUAT) to extend the classical Universal Approximation Theory (UAT) for Transformer-based LLMs. It demonstrates that both Linear and Multi-Head Attention (MHA) components admit matrix-vector representations, enabling Transformer-based architectures to be viewed as DUAT systems with input-dependent parameters. The paper provides formal DUAT formulations for multi-layer Transformers, explains phenomena such as in-context learning, and connects practical techniques like LoRA fine-tuning and pruning to the DUAT framework. By grounding LLM capabilities in dynamic function approximation and contextual interactions, DUAT offers a principled lens for analyzing, designing, and resource-conserving adapting of large language models. The framework also invites future exploration of multimodal and memory-rich DUAT architectures beyond current text-only transformers.

Abstract

Language models have emerged as a critical area of focus in artificial intelligence, particularly with the introduction of groundbreaking innovations like ChatGPT. Large-scale Transformer networks have quickly become the leading approach for advancing natural language processing algorithms. Built on the Transformer architecture, these models enable interactions that closely mimic human communication and, equipped with extensive knowledge, can even assist in guiding human tasks. Despite their impressive capabilities and growing complexity, a key question remains-the theoretical foundations of large language models (LLMs). What makes Transformer so effective for powering intelligent language applications, such as translation and coding? What underlies LLMs' ability for In-Context Learning (ICL)? How does the LoRA scheme enhance the fine-tuning of LLMs? And what supports the practicality of pruning LLMs? To address these critical questions and explore the technological strategies within LLMs, we leverage the Universal Approximation Theory (UAT) to offer a theoretical backdrop, shedding light on the mechanisms that underpin these advancements.

Figures (15)

• Figure 1: The transformation process of the Matrix-Vector Method.
• Figure 1: The process of transforming $\text{Concat}(\hat{\mathbf{H}}_1...\hat{\mathbf{H}}_8)$ in the MHA into its corresponding matrix-vector form $\mathbf{W}_{HV}'\mathbf{x}'=\hat{\mathbf{H}}'$.
• Figure 2: This diagram shows the differences between an UAT and a DUAT.
• Figure 2: Some examples of the DUAT format of multi-layer Transformer. The changes of parameters are represented within the dashed boxes. The parameters on the right of the equations indicate the original values, while those on the left represent the transformed values. There is no specific order of calculation for the parameters within each dashed box, but there is a top-to-bottom calculation order between different dashed boxes.
• Figure 3: The process of converting a linear transformation into its corresponding matrix-vector representation. a: Depicts the general form of a linear transformation. b: Presents a straightforward example of a linear transformation. c: Demonstrates the transformation of the linear operation from b into the matrix-vector format.