Notes- LLM From Scratch (GPT)

GPT architecture looks complex, but if studied properly then everything comes under:

• Metrics and tensors : You should be comfortable handling tensors and understanding their dimensions.
• Probability and Statistics: Softmax, Layer normalization and the multinomial distribution play a very important role in building GPT.
• Calculus: To train GPT, we need to run backpropagation. Chain rule is the core of this.

In my notes I’ve mentioned all the steps involved in building GPT-2 from scratch are as follows:

• Learning about LLM (Large Language Models)
• Stages of building LLM
• Data preprocessing\
• Cleaning and tokenizing text
• Transformer architecture
• Attention mechanisms (Multi Head Attention)
• Coding & Training Model