Hardware Phi-1.5B: A Large Language Model Encodes Hardware Domain Specific Knowledge

TL;DR

The paper tackles the scarcity of hardware-domain data for LLM pretraining by introducing Hardware Phi-1.5B, a $24$-layer Transformer pretrained on a medium-sized hardware-domain dataset with code and hardware-security content. It adopts a four-stage development approach (pretraining, supervised finetuning, instruction-based RL, and in-context deployment) and follows a tiered data strategy (small, medium, large) to build a robust, open-source foundation. Key contributions include the first pretrained hardware-domain LLM, a well-curated three-tier dataset, detailed training methodology, and an evaluation framework, demonstrating potential improvements in hardware design, verification, and security tasks. The work lays groundwork for domain-specific AI in semiconductors, enabling more reliable, efficient AI-assisted hardware pipelines and inviting community collaboration through open sourcing.

Abstract

In the rapidly evolving semiconductor industry, where research, design, verification, and manufacturing are intricately linked, the potential of Large Language Models to revolutionize hardware design and security verification is immense. The primary challenge, however, lies in the complexity of hardware specific issues that are not adequately addressed by the natural language or software code knowledge typically acquired during the pretraining stage. Additionally, the scarcity of datasets specific to the hardware domain poses a significant hurdle in developing a foundational model. Addressing these challenges, this paper introduces Hardware Phi 1.5B, an innovative large language model specifically tailored for the hardware domain of the semiconductor industry. We have developed a specialized, tiered dataset comprising small, medium, and large subsets and focused our efforts on pretraining using the medium dataset. This approach harnesses the compact yet efficient architecture of the Phi 1.5B model. The creation of this first pretrained, hardware domain specific large language model marks a significant advancement, offering improved performance in hardware design and verification tasks and illustrating a promising path forward for AI applications in the semiconductor sector.

Figures (11)

• Figure 1: Four-Stage LLM-Based Assistant Development: Pretraining with raw data for a base model; ideal response-driven supervised fine-tuning; instruction-based reinforcement learning with few-shot examples for a deployable model; culminating in user engagement via in-context learning. The green cells highlight the contributions made in this paper.
• Figure 2: Tokenization example: transform all text to a list of integers.
• Figure 3: Matrix representation of the batch structure in Hardware Phi-1.5B.
• Figure 4: Token prediction probability distribution with preferences after pretrain.
• Figure 5: Visual Representation of Dataset Construction and Segmentation