Transformer Math 101

"Transformer Math 101" by Quentin Anthony, Stella Biderman, and Hailey Schoelkopf presents an overview of the fundamental mathematics guiding the computation and memory usage of transformer models. It highlights how simple equations can be utilised to calculate the training costs, primarily influenced by VRAM requirements, and provides insights into the compute and memory optimisations for efficient training. The post serves as a valuable resource for understanding the scaling laws by OpenAI and DeepMind, and discusses strategies such as mixed-precision training and sharded optimization to manage resource requirements effectively.)

Visit Original Article →