Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Decentralized Autoregressive Generation

Stepan Maschan, Haoxuan Qu, Jun Liu

TL;DR

A theoretical analysis of decentralization of autoregressive generation of autoregressive generation is presented, and the Decentralized Discrete Flow Matching objective is defined, by expressing probability generating velocity as a linear combination of expert flows.

Abstract

We present a theoretical analysis of decentralization of autoregressive generation. We define the Decentralized Discrete Flow Matching objective, by expressing probability generating velocity as a linear combination of expert flows. We also conduct experiments demonstrating the equivalence between decentralized and centralized training settings for multimodal language models across diverse set of benchmarks. Specifically, we compare two distinct paradigms: LLaVA and InternVL 2.5-1B, which uses a fixed CLIP vision encoder and performs full-parameter fine-tuning (ViT+MLP+LLM) during the instruction tuning stage.

Decentralized Autoregressive Generation

TL;DR

A theoretical analysis of decentralization of autoregressive generation of autoregressive generation is presented, and the Decentralized Discrete Flow Matching objective is defined, by expressing probability generating velocity as a linear combination of expert flows.

Abstract

We present a theoretical analysis of decentralization of autoregressive generation. We define the Decentralized Discrete Flow Matching objective, by expressing probability generating velocity as a linear combination of expert flows. We also conduct experiments demonstrating the equivalence between decentralized and centralized training settings for multimodal language models across diverse set of benchmarks. Specifically, we compare two distinct paradigms: LLaVA and InternVL 2.5-1B, which uses a fixed CLIP vision encoder and performs full-parameter fine-tuning (ViT+MLP+LLM) during the instruction tuning stage.
Paper Structure (21 sections, 1 theorem, 24 equations, 9 tables)

This paper contains 21 sections, 1 theorem, 24 equations, 9 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Continuous Time Discrete Flow Matching
  4. Theoretical Analysis
  5. Discrete Time Discrete Flow Matching
  6. Autoregressive sampling
  7. Decentralization
  8. Method
  9. Expert training
  10. Data partitioning
  11. Inference strategy
  12. Experiments
  13. LLaVA
  14. InternVL
  15. Evaluation
  16. ...and 6 more sections

Key Result

Theorem 1

Probability generating velocity $u_t^i(x^i)$ defined as follows: generates the probability path $p_t(x)$

Theorems & Definitions (1)

  • Theorem 1