Current AI architectures

A clear, theory-first walkthrough of the architectures driving today’s generative AI. We focus on how and why these models work. You’ll leave with a crisp mental map of Transformers, Diffusion models, and State-Space Models, plus the vocabulary to read papers and evaluate trade-offs in the real world.

Pillar I: Foundations of Modern Scale

• Pretraining and transfer learning: why scale works.
• Tokens, context windows, and inference constraints.

Pillar II: The Transformer

• Self-attention: Q/K/V, multi-head attention, residuals, layer norm.
• Positional information: absolute, RoPE, ALiBi.
• Model blueprints: BERT (encoder), GPT (decoder), T5 (encoder–decoder).
• Efficiency patterns: MQA/GQA, KV caching, sliding windows.

Pillar III: Multimodality and Retrieval

• Vision Transformer (ViT) and contrastive learning with CLIP
• Retrieval-Augmented Generation (RAG): when and why to add external knowledge

Pillar IV: Beyond Attention

• State-Space Models (S4/S5, Mamba): linear recurrence and selective state updates
• Other lines of work: RWKV, Hyena, RetNet (high-level intuition)
• Mixture-of-Experts (MoE): routing, capacity vs. communication trade-offs

Pillar V: Diffusion, Brief and Practical

• DDPM basics: forward noising and learned denoising
• DiT (Diffusion Transformer): why Transformers show up here too

Epilogue: What’s Next

• Longer context, tool use and agents, modular systems, energy efficiency

Outcomes:

-Unified mental model: How Transformers, Diffusion, and SSMs differ from CNNs/RNNs and when to reach for each family.

-Paper fluency: Read model cards and papers with confidence (positional encodings, attention variants, SSMs, MoE).

-Practical judgment: Understand trade-offs among context length, latency, memory footprint, and scaling.