Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

oboro: Text-to-Image Synthesis on Limited Data using Flow-based Diffusion Transformer with MMH Attention

Ryusuke Mizutani, Kazuaki Matano, Tsugumi Kadowaki, Haruki Tenya, Layris, nuigurumi, Koki Hashimoto, Yu Tanaka

TL;DR

oboro addresses the demand for high-quality text-to-image synthesis under limited, rights-cleared data in anime production. It introduces a diffusion-model framework built from scratch that uses an I-CFM flow and a Diffusion Transformer with Multi-Multi-Head Attention, paired with a T5 XXL text encoder and a FLUX 16-channel VAE. The model achieves competitive image quality with far less data and compute than SD1.5/SDXL, enabling practical use in commercial anime workflows, and is released as open-source to support Japan's AI ecosystem. A two-stage training strategy (foundation on rights-cleared data, followed by company-specific fine-tuning) further enhances adaptability while mitigating copyright risks.

Abstract

This project was conducted as a 2nd-term adopted project of the "Post-5G Information and Communication System Infrastructure Enhancement R&D Project Development of Competitive Generative AI Foundation Models (GENIAC)," a business of the Ministry of Economy, Trade and Industry (METI) and the New Energy and Industrial Technology Development Organization (NEDO). To address challenges such as labor shortages in Japan's anime production industry, this project aims to develop an image generation model from scratch. This report details the technical specifications of the developed image generation model, "oboro:." We have developed "oboro:," a new image generation model built from scratch, using only copyright-cleared images for training. A key characteristic is its architecture, designed to generate high-quality images even from limited datasets. The foundation model weights and inference code are publicly available alongside this report. This project marks the first release of an open-source, commercially-oriented image generation AI fully developed in Japan. AiHUB originated from the OSS community; by maintaining transparency in our development process, we aim to contribute to Japan's AI researcher and engineer community and promote the domestic AI development ecosystem.

oboro: Text-to-Image Synthesis on Limited Data using Flow-based Diffusion Transformer with MMH Attention

TL;DR

oboro addresses the demand for high-quality text-to-image synthesis under limited, rights-cleared data in anime production. It introduces a diffusion-model framework built from scratch that uses an I-CFM flow and a Diffusion Transformer with Multi-Multi-Head Attention, paired with a T5 XXL text encoder and a FLUX 16-channel VAE. The model achieves competitive image quality with far less data and compute than SD1.5/SDXL, enabling practical use in commercial anime workflows, and is released as open-source to support Japan's AI ecosystem. A two-stage training strategy (foundation on rights-cleared data, followed by company-specific fine-tuning) further enhances adaptability while mitigating copyright risks.

Abstract

This project was conducted as a 2nd-term adopted project of the "Post-5G Information and Communication System Infrastructure Enhancement R&D Project Development of Competitive Generative AI Foundation Models (GENIAC)," a business of the Ministry of Economy, Trade and Industry (METI) and the New Energy and Industrial Technology Development Organization (NEDO). To address challenges such as labor shortages in Japan's anime production industry, this project aims to develop an image generation model from scratch. This report details the technical specifications of the developed image generation model, "oboro:." We have developed "oboro:," a new image generation model built from scratch, using only copyright-cleared images for training. A key characteristic is its architecture, designed to generate high-quality images even from limited datasets. The foundation model weights and inference code are publicly available alongside this report. This project marks the first release of an open-source, commercially-oriented image generation AI fully developed in Japan. AiHUB originated from the OSS community; by maintaining transparency in our development process, we aim to contribute to Japan's AI researcher and engineer community and promote the domestic AI development ecosystem.

Paper Structure

This paper contains 79 sections, 10 figures, 1 table.

Table of Contents

  1. Development Background
  2. Improvement of Image Generation Technology
  3. Ethical Issues and Dataset Constraints in Image Generation AI
  4. Performance and Training Dataset Requirements for Image Generation AI Introduced in Anime Production
  5. Status of Competing Models
  6. Text Encoders: Deepening Semantic Understanding
  7. CLIP: Achieving Conceptual Alignment
  8. T5: Precise Interpretation of Language Structure
  9. Gemma and Other Encoders
  10. Image Inference Architecture: From U-Net to Transformer
  11. U-Net: The Foundation of Convolutional Networks
  12. DiT (Diffusion Transformer): Achieving Scalability
  13. VAE: Representation and Reconstruction in Latent Space
  14. The Role of VAE in Latent Diffusion Models
  15. The Trade-off between Compression Rate and Detail Preservation
  16. ...and 64 more sections

Figures (10)

  • Figure 1: Our Model: Overall Structure. Inputs (time, text, image) are processed by encoders, combined, passed through N Blocks, and decoded by the VAE decoder.
  • Figure 2: Block Structure, showing Sandwiched Self-Attention and Sandwiched MLP with residual connections.
  • Figure 3: MLP Block Structure, showing Linear layers, SiLU activation, and element-wise multiplication.
  • Figure 4: Self-Attention Structure, showing inputs (position, vector), Linear layers, Layer Norm (LN), SDPA, and position modulation.
  • Figure 5: Sandwich Normalization Structure, showing RMS, scaling, and conditioning.
  • ...and 5 more figures