Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Whack-a-Chip: The Futility of Hardware-Centric Export Controls

Ritwik Gupta, Leah Walker, Andrew W. Reddie

TL;DR

It is argued that Tencent's ability to power its Hunyuan-Large model with non-export controlled NVIDIA H20s exemplifies broader gains in efficiency in machine learning that have eroded the moat that the United States initially built via its existing export controls.

Abstract

U.S. export controls on semiconductors are widely known to be permeable, with the People's Republic of China (PRC) steadily creating state-of-the-art artificial intelligence (AI) models with exfiltrated chips. This paper presents the first concrete, public evidence of how leading PRC AI labs evade and circumvent U.S. export controls. We examine how Chinese companies, notably Tencent, are not only using chips that are restricted under U.S. export controls but are also finding ways to circumvent these regulations by using software and modeling techniques that maximize less capable hardware. Specifically, we argue that Tencent's ability to power its Hunyuan-Large model with non-export controlled NVIDIA H20s exemplifies broader gains in efficiency in machine learning that have eroded the moat that the United States initially built via its existing export controls. Finally, we examine the implications of this finding for the future of the United States' export control strategy.

Whack-a-Chip: The Futility of Hardware-Centric Export Controls

TL;DR

It is argued that Tencent's ability to power its Hunyuan-Large model with non-export controlled NVIDIA H20s exemplifies broader gains in efficiency in machine learning that have eroded the moat that the United States initially built via its existing export controls.

Abstract

U.S. export controls on semiconductors are widely known to be permeable, with the People's Republic of China (PRC) steadily creating state-of-the-art artificial intelligence (AI) models with exfiltrated chips. This paper presents the first concrete, public evidence of how leading PRC AI labs evade and circumvent U.S. export controls. We examine how Chinese companies, notably Tencent, are not only using chips that are restricted under U.S. export controls but are also finding ways to circumvent these regulations by using software and modeling techniques that maximize less capable hardware. Specifically, we argue that Tencent's ability to power its Hunyuan-Large model with non-export controlled NVIDIA H20s exemplifies broader gains in efficiency in machine learning that have eroded the moat that the United States initially built via its existing export controls. Finally, we examine the implications of this finding for the future of the United States' export control strategy.

Paper Structure

This paper contains 15 sections, 4 figures.

Table of Contents

  1. Introduction
  2. Chinese Firms are Outpacing Hardware Restrictions
  3. Hunyuan-Large: A SOTA, Open-Source LLM
  4. Efficiently Training SOTA Models on Nerfed Hardware with Better Software
  5. Ensembles and Mixtures-of-Experts
  6. Mixed-Precision Training via bfloat16
  7. Quantization
  8. Large VRAM and Sharded Training
  9. Effectively Using Networked GPUs
  10. Overriding Software Limitations in Hardware
  11. Chinese Firms Bypass Chip Restrictions
  12. Tracking the Use of Export-Controlled Chips Through Code Signatures
  13. Avenues of Access
  14. A Reckoning for U.S. Semiconductor Export Controls
  15. Conclusion

Figures (4)

  • Figure 1: A non-comprehensive list of GPUs under consideration with the potential to train AI models.
  • Figure 2: The use of NCCL eliminates AMD GPUs from consideration entirely.
  • Figure 3: bfloat16 is only available beginning with the Ampere micro-architecture.
  • Figure 4: GPUDirect RDMA is a feature that NVIDIA limits to data center GPUs.