The Breeze 2 Herd of Models: Traditional Chinese LLMs Based on Llama with Vision-Aware and Function-Calling Capabilities

TL;DR

Breeze2 addresses the underrepresentation of Traditional Chinese in large language models by building two multi-modal LLMs (3B and 8B) based on Llama with vision-aware and function-calling capabilities. It employs a three-stage training pipeline—extended text-to-text pre-training, vision alignment, and post-training for instruction following, visual instruction, and function calling—using a 900 GB Traditional Chinese/Taiwan corpus and diverse multimodal datasets. The model demonstrates strong Taiwan-specific knowledge, robust long-context handling, competitive function-calling performance, and solid vision-language capabilities, including on-device deployment via a mobile app. These results indicate Breeze2’s potential for Taiwan-focused AI applications and highlight pathways for further improvements in vision-language fusion, mobile efficiency, and scaling to larger models.

Abstract

Llama-Breeze2 (hereinafter referred to as Breeze2) is a suite of advanced multi-modal language models, available in 3B and 8B parameter configurations, specifically designed to enhance Traditional Chinese language representation. Building upon the Llama 3.2 model family, we continue the pre-training of Breeze2 on an extensive corpus to enhance the linguistic and cultural heritage of Traditional Chinese. In addition to language modeling capabilities, we significantly augment the models with function calling and vision understanding capabilities. At the time of this publication, as far as we are aware, absent reasoning-inducing prompts, Breeze2 are the strongest performing models in Traditional Chinese function calling and image understanding in its size class. The effectiveness of Breeze2 is benchmarked across various tasks, including Taiwan general knowledge, instruction-following, long context, function calling, and vision understanding. We are publicly releasing all Breeze2 models under the Llama 3.2 Community License. We also showcase the capabilities of the model running on mobile platform with a mobile application which we also open source.

Figures (4)

• Figure 1: Breeze2 Architecture. Breeze2 utilizes the widely-adopted “ViT-MLP-LLM” paradigm, which integrates a pre-trained InternViT-300M-448px with LLMs of various sizes through a vision-to-language MLP projector. Specifically, it combines the InternViT-300M-448px with Llama 3.2 3B and Llama 3.1 8B models. This architecture leverages the strengths of ViT, MLP, and LLM to process both visual and textual inputs effectively.
• Figure 2: Training Process of Breeze2 models. Breeze2 models were trained with a multi-stage training approach, starting from the Llama 3.2 3B and Llama 3.1 8B base models. The process starts with extended text-to-text pretraining to enhance Traditional Chinese language understanding. It is followed by vision-alignment pretraining to integrate visual features, and post-training for refining text and visual instruction tuning and function calling capabilities.
• Figure 3: Image for demo showing a prize distribution table.
• Figure 4: Overview of the App Interface Design. This figure illustrates the main user interaction elements in the application. Users can select a photo from the gallery or capture a real-time picture to initiate visual processing. A dedicated input text region enables users to provide textual commands. Additionally, the application integrates a microphone for automatic speech recognition (ASR) to convert speech into text, allowing for hands-free interaction. The system also supports text-to-speech (TTS) functionality, which vocalizes generated content back to the user, ensuring accessibility and an enhanced user experience.