Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Training a Huggingface Model on AWS Sagemaker (Without Tears)

Liling Tan

TL;DR

This paper tackles the barrier researchers face when training HuggingFace models on AWS SageMaker due to fragmented documentation and cloud onboarding. It presents a practical demo that consolidates essential steps, from domain setup and notebook provisioning to HuggingFace estimator configuration and code-location strategies. Key contributions include a step-by-step walkthrough of HuggingFace estimator usage, methods to handle environment dependencies via SAGEMAKER_ENVIRONMENT, and guidance on quota management and hyperparameter tuning workflows. The work aims to democratize cloud adoption and foster community engagement through canonical forum discussions and open-source appendix. The approach has practical impact by enabling rapid, accessible cloud-based experimentation with HuggingFace Transformers on SageMaker.

Abstract

The development of Large Language Models (LLMs) has primarily been driven by resource-rich research groups and industry partners. Due to the lack of on-premise computing resources required for increasingly complex models, many researchers are turning to cloud services like AWS SageMaker to train Hugging Face models. However, the steep learning curve of cloud platforms often presents a barrier for researchers accustomed to local environments. Existing documentation frequently leaves knowledge gaps, forcing users to seek fragmented information across the web. This demo paper aims to democratize cloud adoption by centralizing the essential information required for researchers to successfully train their first Hugging Face model on AWS SageMaker from scratch.

Training a Huggingface Model on AWS Sagemaker (Without Tears)

TL;DR

This paper tackles the barrier researchers face when training HuggingFace models on AWS SageMaker due to fragmented documentation and cloud onboarding. It presents a practical demo that consolidates essential steps, from domain setup and notebook provisioning to HuggingFace estimator configuration and code-location strategies. Key contributions include a step-by-step walkthrough of HuggingFace estimator usage, methods to handle environment dependencies via SAGEMAKER_ENVIRONMENT, and guidance on quota management and hyperparameter tuning workflows. The work aims to democratize cloud adoption and foster community engagement through canonical forum discussions and open-source appendix. The approach has practical impact by enabling rapid, accessible cloud-based experimentation with HuggingFace Transformers on SageMaker.

Abstract

The development of Large Language Models (LLMs) has primarily been driven by resource-rich research groups and industry partners. Due to the lack of on-premise computing resources required for increasingly complex models, many researchers are turning to cloud services like AWS SageMaker to train Hugging Face models. However, the steep learning curve of cloud platforms often presents a barrier for researchers accustomed to local environments. Existing documentation frequently leaves knowledge gaps, forcing users to seek fragmented information across the web. This demo paper aims to democratize cloud adoption by centralizing the essential information required for researchers to successfully train their first Hugging Face model on AWS SageMaker from scratch.
Paper Structure (14 sections, 9 figures)

This paper contains 14 sections, 9 figures.

Table of Contents

  1. Introduction
  2. AWS Sagemaker
  3. Amazon SageMaker > Domains > Domain: mtsdemo > Jupyter Notebook
  4. Amazon SageMaker > Training Jobs > Model Training
  5. Your First Huggingface Script on AWS Sagemaker
  6. HuggingFace Estimator
  7. Specifying Code Location for HuggingFace Estimator
  8. Come for the Code, Stay for the Community
  9. Related Work
  10. Conclusion
  11. Limitations and Risk
  12. Controlling AWS Instance for HuggingFace Estimator
  13. Selecting Docker Image for HuggingFace Estimator
  14. Amazon SageMaker > Hyperparameter tuning jobs

Figures (9)

  • Figure 1: Sample code to train a Huggingface model using AWS Sagemaker
  • Figure 2: Shutting down virtual machine instances hosting the Jupyter notebook(s)
  • Figure 3: Monitoring Model Trainings on the AWS Sagemaker Console
  • Figure 4: Working Sample Code to Fine-tune a Translation model with Huggingface Transformers
  • Figure 5: Working Sample Code to do Hyperparameter Tuning for a Translation model with Huggingface Transformers
  • ...and 4 more figures