01 - Foundations: Large Language Models
Estimated time to read: 5 minutes
Welcome to the Foundations of Large Language Models module. This module explores the transition from mathematical operations to emergent AI phenomena, focusing on subword segmentation, vector spaces, and latent manifold representations.
🧭 Module Roadmap & Laboratories
To keep the material focused and structured, we have divided this module into four distinct, comprehensive laboratories:
1. 📊 LAB-01: Tokenization Math of Compression
Benchmarks BPE vs WordPiece tokenizers, analyzes the character footprint of structured data (JSON), and quantifies the multilingual "Token Tax" on non-English text.
- Core concepts: subword segmentation, compression ratio, byte representation.
- Interactive Playground: tokenization_playground.ipynb
2. 📐 LAB-02: Embedding Geometry & Semantic Search
Explores the Manifold Hypothesis in high-dimensional vector spaces, implements Cosine Similarity manually, and visualizes word clusters in 2D and 3D.
- Core concepts: metric spaces, cosine vs. Euclidean distances, PCA, t-SNE, semantic search indexing.
- Interactive Playground: embeddings_playground.ipynb
3. 🧠 LAB-03: Attention from Scratch: PyTorch Linear Algebra
Implements scaled dot-product attention and multi-head attention from scratch using PyTorch linear algebra. Computes bidirectional vs. causal masked matrices.
- Core concepts: Scaled Dot-Product, Multi-Head Attention, causal masking, KV projection.
- Interactive Playground: attention_playground.ipynb
4. 🎲 LAB-04: Inference Math: Sampling, Quantization & Loss Analysis
Implements autoregressive token sampling strategies (Temperature, Top-k, Top-p), evaluates Cross-Entropy Loss & Perplexity, and quantifies GPU VRAM & KV-Cache memory footprint across FP16, INT8, and INT4 quantization schemes.
- Core concepts: Temperature scaling, Top-k/Top-p Nucleus sampling, Perplexity, Int8/Int4 quantization math, KV Cache memory overhead.
- Interactive Playground: inference_playground.ipynb
🛠️ Resources & References
Below are the academic references and technical guides used in this module, styled in a technical post-card format:
📄 Academic & Seminal Papers
Attention Is All You Need
The seminal paper introducing the Transformer architecture, replacing recurrent and convolutional neural networks with self-attention mechanism layers.
Neural Machine Translation of Rare Words with Subword Units
The original paper adapting Byte Pair Encoding (BPE) for word segmentation in machine translation, solving the out-of-vocabulary words problem.
Visualizing Data using t-SNE
The landmark paper introducing t-Distributed Stochastic Neighbor Embedding, showcasing its ability to preserve local neighborhood topologies compared to linear methods.
📖 Technical References & Guides
Hugging Face Tokenizers Guide
Practical implementation details, benchmarks, and algorithms for BPE, WordPiece, and SentencePiece tokenizers.
Ollama & LiteLLM Integration Docs
Guide on running local LLM inference engines and wrapping them with OpenAI-compatible routing for development fallbacks.
The Manifold Hypothesis (Wikipedia)
Deep dive into why high-dimensional data distributions tend to concentrate near lower-dimensional, non-linear manifolds, which forms the mathematical foundation of neural embeddings.
Principal Component Analysis (PCA)
Historical paper and mathematical proof detailing the process of projecting high-dimensional points onto orthogonal eigenvectors to maximize variance.
Cosine Similarity & Vector Spaces
Overview of cosine similarity, its mathematical formulation, and its application in high-dimensional information retrieval and metric vector spaces.
The Curious Case of Neural Text Degeneration
Landmark paper introducing Nucleus (Top-p) sampling, demonstrating how dynamic probability truncation prevents repetitive loops in neural text generation.
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
Introduces vector-wise quantization and outlier feature extraction enabling 8-bit transformer matrix multiplication with zero performance degradation.