CATransformers: Carbon Aware Transformers Through Joint Model-Hardware Optimization

TL;DR

CATransformers presents a carbon-aware co-design framework that jointly optimizes Transformer architectures and edge accelerators to minimize total carbon emissions, accounting for both operational and embodied carbon. It blends a multi-objective Bayesian optimization loop, an accuracy proxy evaluator, and a hardware carbon estimator to explore large model-hardware spaces, yielding carbon reductions up to $30\%$ while maintaining performance. The CarbonCLIP case study shows multi-modal models can achieve lower carbon footprints with comparable accuracy, illustrating practical applicability to real-world edge deployments. This work highlights the importance of holistic sustainability metrics in AI design and provides an open-source toolchain to enable reproducible, carbon-aware optimization for future edge AI systems.

Abstract

Machine learning solutions are rapidly adopted to enable a variety of key use cases, from conversational AI assistants to scientific discovery. This growing adoption is expected to increase the associated lifecycle carbon footprint, including both \emph{operational carbon} from training and inference and \emph{embodied carbon} from AI hardware manufacturing. We introduce \ourframework -- the first carbon-aware co-optimization framework for Transformer-based models and hardware accelerators. By integrating both operational and embodied carbon into early-stage design space exploration, \ourframework enables sustainability-driven model architecture and hardware accelerator co-design that reveals fundamentally different trade-offs than latency- or energy-centric approaches. Evaluated across a range of Transformer models, \ourframework consistently demonstrates the potential to reduce total carbon emissions -- by up to 30\% -- while maintaining accuracy and latency. We further highlight its extensibility through a focused case study on multi-modal models. Our results emphasize the need for holistic optimization methods that prioritize carbon efficiency without compromising model capability and execution time performance. The source code of \ourframework is available at {\small{\href{https://github.com/facebookresearch/CATransformers}{\texttt{https://github.com/facebookresearch/CATransformers}}}}.

Figures (14)

• Figure 1: CarbonCLIP models achieve lower carbon footprint and higher accuracy compared to baseline CLIP models.
• Figure 2: Overview of the CATransformers framework. The Bayesian optimizer iteratively explores model and hardware configurations using accuracy, carbon, and latency estimates from evaluation modules, outputting optimized co-designs.
• Figure 3: Overview of model pruning dimensions and hardware template for CATransformers
• Figure 4: ISO-Accuracy plot showing the latency–carbon trade-off across optimization strategies, with accuracy matched within ±1%.
• Figure 5: Pareto frontiers under varying compute constraints.