Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Kandinsky 3.0 Technical Report

Vladimir Arkhipkin, Andrei Filatov, Viacheslav Vasilev, Anastasia Maltseva, Said Azizov, Igor Pavlov, Julia Agafonova, Andrey Kuznetsov, Denis Dimitrov

TL;DR

Kandinsky 3.0 introduces a large latent-diffusion text-to-image model with 11.9B parameters, improving text understanding and image fidelity via a single-stage pipeline that directly uses text embeddings. The architecture integrates a frozen text encoder, a deep U-Net, and a Sber-MoVQGAN decoder, trained on a diverse, multi-resolution dataset with staged pretraining. It also launches Kandinsky 3.1 distillation for fast 4-step inference, Kandinsky SuperRes for 4K generation, and a prompt-beautification flow using an LLM, while expanding capabilities to inpainting, image editing, image-to-video, and text-to-video. The work is complemented by extensive human evaluation, comparisons against multiple baselines, and a discussion of limitations and ethical considerations, with code and weights released to promote openness and reproducibility.

Abstract

We present Kandinsky 3.0, a large-scale text-to-image generation model based on latent diffusion, continuing the series of text-to-image Kandinsky models and reflecting our progress to achieve higher quality and realism of image generation. In this report we describe the architecture of the model, the data collection procedure, the training technique, and the production system for user interaction. We focus on the key components that, as we have identified as a result of a large number of experiments, had the most significant impact on improving the quality of our model compared to the others. We also describe extensions and applications of our model, including super resolution, inpainting, image editing, image-to-video generation, and a distilled version of Kandinsky 3.0 - Kandinsky 3.1, which does inference in 4 steps of the reverse process and 20 times faster without visual quality decrease. By side-by-side human preferences comparison, Kandinsky becomes better in text understanding and works better on specific domains. The code is available at https://github.com/ai-forever/Kandinsky-3

Kandinsky 3.0 Technical Report

TL;DR

Kandinsky 3.0 introduces a large latent-diffusion text-to-image model with 11.9B parameters, improving text understanding and image fidelity via a single-stage pipeline that directly uses text embeddings. The architecture integrates a frozen text encoder, a deep U-Net, and a Sber-MoVQGAN decoder, trained on a diverse, multi-resolution dataset with staged pretraining. It also launches Kandinsky 3.1 distillation for fast 4-step inference, Kandinsky SuperRes for 4K generation, and a prompt-beautification flow using an LLM, while expanding capabilities to inpainting, image editing, image-to-video, and text-to-video. The work is complemented by extensive human evaluation, comparisons against multiple baselines, and a discussion of limitations and ethical considerations, with code and weights released to promote openness and reproducibility.

Abstract

We present Kandinsky 3.0, a large-scale text-to-image generation model based on latent diffusion, continuing the series of text-to-image Kandinsky models and reflecting our progress to achieve higher quality and realism of image generation. In this report we describe the architecture of the model, the data collection procedure, the training technique, and the production system for user interaction. We focus on the key components that, as we have identified as a result of a large number of experiments, had the most significant impact on improving the quality of our model compared to the others. We also describe extensions and applications of our model, including super resolution, inpainting, image editing, image-to-video generation, and a distilled version of Kandinsky 3.0 - Kandinsky 3.1, which does inference in 4 steps of the reverse process and 20 times faster without visual quality decrease. By side-by-side human preferences comparison, Kandinsky becomes better in text understanding and works better on specific domains. The code is available at https://github.com/ai-forever/Kandinsky-3
Paper Structure (23 sections, 1 equation, 43 figures, 3 tables)

This paper contains 23 sections, 1 equation, 43 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Interaction System
  3. Kandinsky 3.0 Architecture
  4. Overall pipeline
  5. U-Net architecture
  6. Text encoder
  7. Sber-MoVQGAN
  8. Data and Training Strategy
  9. Data.
  10. Training.
  11. Additional Features
  12. Kandinsky 3.1 (aka Kandinsky 3.0 Distilled)
  13. Kandinsky SuperRes
  14. Prompt beautification
  15. Applications
  16. ...and 8 more sections

Figures (43)

  • Figure 1: Kandinsky 3.0 overall pipeline architecture. It consists of a text encoder, a latent conditioned diffusion model, and an image decoder.
  • Figure 2: Kandinsky 3.0 U-Net architecture. The architecture is based on modified BigGAN-deep blocks (left and right -- downsample and upsample versions), which allows us to increase the depth of the architecture due to the presence of bottlenecks. The attention layers are arranged at levels with a lower resolution than the original image.
  • Figure 3: Discriminator architecture for Kandinsky 3.1. Gray blocks inherit the weight of Kandinsky 3.0 and remain frozen during training.
  • Figure 4: Comparison of Kandinsky SuperRes, Stable Diffusion and Real-ESRGAN models at 1024 resolution. Better to zoom image.
  • Figure 5: Example of Kandinsky SuperRes generation in 4K resolution. Better to zoom image.
  • ...and 38 more figures