Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ChipNeMo: Domain-Adapted LLMs for Chip Design

Mingjie Liu, Teodor-Dumitru Ene, Robert Kirby, Chris Cheng, Nathaniel Pinckney, Rongjian Liang, Jonah Alben, Himyanshu Anand, Sanmitra Banerjee, Ismet Bayraktaroglu, Bonita Bhaskaran, Bryan Catanzaro, Arjun Chaudhuri, Sharon Clay, Bill Dally, Laura Dang, Parikshit Deshpande, Siddhanth Dhodhi, Sameer Halepete, Eric Hill, Jiashang Hu, Sumit Jain, Ankit Jindal, Brucek Khailany, George Kokai, Kishor Kunal, Xiaowei Li, Charley Lind, Hao Liu, Stuart Oberman, Sujeet Omar, Ghasem Pasandi, Sreedhar Pratty, Jonathan Raiman, Ambar Sarkar, Zhengjiang Shao, Hanfei Sun, Pratik P Suthar, Varun Tej, Walker Turner, Kaizhe Xu, Haoxing Ren

TL;DR

ChipNeMo demonstrates that domain-adaptive pretraining combined with domain-adaptive tokenization, targeted model alignment, and retrieval-augmented generation can significantly boost LLM performance on chip design tasks. The largest model (ChipNeMo-70B) matches or exceeds GPT-4 on several tasks, while requiring only a small fraction of full pretraining compute. Across engineering assistant chatbot, EDA script generation, and bug analysis, domain adaptation improves grounding, code quality, and summarization clarity, with substantial gains in RAG-grounded responses. These findings offer a practical pathway to cost-effective, domain-specific LLM deployment in hardware design ecosystems.

Abstract

ChipNeMo aims to explore the applications of large language models (LLMs) for industrial chip design. Instead of directly deploying off-the-shelf commercial or open-source LLMs, we instead adopt the following domain adaptation techniques: domain-adaptive tokenization, domain-adaptive continued pretraining, model alignment with domain-specific instructions, and domain-adapted retrieval models. We evaluate these methods on three selected LLM applications for chip design: an engineering assistant chatbot, EDA script generation, and bug summarization and analysis. Our evaluations demonstrate that domain-adaptive pretraining of language models, can lead to superior performance in domain related downstream tasks compared to their base LLaMA2 counterparts, without degradations in generic capabilities. In particular, our largest model, ChipNeMo-70B, outperforms the highly capable GPT-4 on two of our use cases, namely engineering assistant chatbot and EDA scripts generation, while exhibiting competitive performance on bug summarization and analysis. These results underscore the potential of domain-specific customization for enhancing the effectiveness of large language models in specialized applications.

ChipNeMo: Domain-Adapted LLMs for Chip Design

TL;DR

ChipNeMo demonstrates that domain-adaptive pretraining combined with domain-adaptive tokenization, targeted model alignment, and retrieval-augmented generation can significantly boost LLM performance on chip design tasks. The largest model (ChipNeMo-70B) matches or exceeds GPT-4 on several tasks, while requiring only a small fraction of full pretraining compute. Across engineering assistant chatbot, EDA script generation, and bug analysis, domain adaptation improves grounding, code quality, and summarization clarity, with substantial gains in RAG-grounded responses. These findings offer a practical pathway to cost-effective, domain-specific LLM deployment in hardware design ecosystems.

Abstract

ChipNeMo aims to explore the applications of large language models (LLMs) for industrial chip design. Instead of directly deploying off-the-shelf commercial or open-source LLMs, we instead adopt the following domain adaptation techniques: domain-adaptive tokenization, domain-adaptive continued pretraining, model alignment with domain-specific instructions, and domain-adapted retrieval models. We evaluate these methods on three selected LLM applications for chip design: an engineering assistant chatbot, EDA script generation, and bug summarization and analysis. Our evaluations demonstrate that domain-adaptive pretraining of language models, can lead to superior performance in domain related downstream tasks compared to their base LLaMA2 counterparts, without degradations in generic capabilities. In particular, our largest model, ChipNeMo-70B, outperforms the highly capable GPT-4 on two of our use cases, namely engineering assistant chatbot and EDA scripts generation, while exhibiting competitive performance on bug summarization and analysis. These results underscore the potential of domain-specific customization for enhancing the effectiveness of large language models in specialized applications.
Paper Structure (37 sections, 18 figures, 16 tables)

This paper contains 37 sections, 18 figures, 16 tables.

Table of Contents

  1. Introduction
  2. ChipNeMo Domain Adaptation Methods
  3. Domain-Adaptive Tokenization
  4. Domain-Adaptive Tokenization
  5. Domain Adaptive Pretraining
  6. Model Alignment
  7. Domain-Adapted Retrieval Model
  8. Evaluations
  9. Domain-Adaptive Tokenization
  10. Domain Adaptive Pretraining
  11. Training Ablation Studies
  12. Training Cost
  13. RAG and Engineering Assistant Chatbot
  14. EDA Script Generation
  15. Bug Summarization and Analysis
  16. ...and 22 more sections

Figures (18)

  • Figure 1: ChipNeMo Training Flow
  • Figure 2: Smoothed Training Loss for ChipNeMo with Tokenizer Augmentation.
  • Figure 3: RAG Implementation Variations
  • Figure 4: Domain-Adapted ChipNeMo Tokenizer Improvements.
  • Figure 5: Chip Domain Benchmark Result for ChipNeMo.
  • ...and 13 more figures