Zero to AI engineer, organized by topic.

Why first: every later track assumes array-native thinking. Loop-based code is the single most common beginner habit that has to be unlearned before anything else sticks.

Covers: why array operations beat loops, array shapes, broadcasting rules, DataFrame slicing with .loc, boolean masks, groupby.

Why it matters later: the matrix multiply you hand-code here is the same operation you'll tile into a CUDA kernel in the GPU track and feed to tensor cores after that — it is genuinely the same 20 lines, at increasing levels of "how fast can this go."

Covers: vectors as arrows and as data, span, linear transformations, matrix multiplication as composed transformations.

Why it matters later: the chain rule is backpropagation — not an analogy for it, the literal mechanism. Skimming this is the single most common reason "my net trains but I don't know why" happens later.

Covers: derivative as sensitivity to a nudge; the chain rule; partial derivatives — the gradient is a vector of partials, one per input (the source curriculum flags this explicitly as the jump from single-variable to multivariable calculus that neural nets require).

Covers: distributions, mean and variance, sampling, the normal distribution, and why ML treats data as draws from a distribution.

Covers: training vs. testing, the bias/variance tradeoff, overfitting.

Gradient descent end to end: loss surface, learning rate, convergence — no new material, this is purely a build-and-consolidate stretch.

Why it matters later: softmax reappears inside attention, inside LLM sampling, and inside every classifier after this — the source curriculum is explicit that this is the one place to learn it properly, once.

Covers: decision boundaries, the sigmoid, log loss, odds and log-odds, softmax as sigmoid's multi-class generalization, and why cross-entropy is softmax's natural loss.

Covers: decision trees (splits, impurity), why forests reduce variance, the boosting idea.

Covers: cross-validation, the confusion-matrix family, ROC/AUC, precision vs. recall, and data leakage — described in the source material as "the silent killer" of otherwise-plausible model results.

Covers: encoding categoricals, scaling, imputation, leakage-safe pipelines.

Covers: k-means step by step, hierarchical clustering, principal component analysis.

No new material — full-cycle consolidation: EDA, cleaning, feature engineering, model comparison, cross-validated scoring, then a real leaderboard submission.

Covers: what a network is, layers as learned transformations, gradient descent on a loss landscape.

Why it's the anchor of this whole track: the source curriculum calls this "the single most important session of the program" — building the autograd engine yourself is what makes every later framework (PyTorch's autograd, and eventually your own custom CUDA ops) legible rather than magical.

Build a minimal autograd Value class with backward passes for addition, multiplication, and tanh; then a tiny neural-net library (Neuron, Layer, MLP) with a training loop on toy data.

Covers: tensors, autograd, nn.Module, Dataset/DataLoader, the canonical training loop.

Covers: loss functions, SGD vs. Adam, learning-rate effects, recognizing overfitting in training curves, and the standard anti-overfitting toolkit — dropout, weight decay (L2), early stopping — and what each actually does to the optimization, not just that it "helps."

Covers: convolutions as learned filters, pooling, parameter sharing, why CNNs fit image data specifically.

No new material — consolidation and a real image-classification project.

Why it's not as new as it looks: the source curriculum points out explicitly that the core of attention is matrix multiplication (Track 1) plus softmax (Track 2) — nothing you haven't already built by hand.

Covers: why fixed-window and recurrent approaches struggle at long range; attention as a learned, weighted lookup across a sequence.

Covers: next-token prediction as the entire training objective; sampling; a bigram and then an MLP-based character-level language model.

Why it's here: this is the gap between "my net trains" and "I know why my net trains" — and LayerNorm, BatchNorm's cousin, sits inside every transformer block you build next.

Covers: activation statistics, dead neurons, vanishing/exploding gradients, why weight initialization matters, and BatchNorm.

Covers: Byte Pair Encoding from scratch, encode/decode, and why several of an LLM's famous weaknesses — spelling, arithmetic, non-English text — are really tokenizer artifacts rather than reasoning failures. Tokenizers are a separately-trained component with their own training set, a point most introductory material glosses over.

The transformer block, multi-head self-attention, residual connections, LayerNorm, and the full training loop — assembled from every primitive built in Tracks 1–4 so far.

Covers: pretraining vs. finetuning, how a raw next-token predictor becomes a helpful assistant (supervised fine-tuning, RLHF), scaling laws, tool use, and LLM security — the pipeline stages most curricula mention but skip explaining.

Covers: the hosted-model ecosystem and Transformers-library basics; system prompts, few-shot examples, JSON-constrained/structured output, retries and timeouts.

Huyen deepening: Chapter 5, "Prompt Engineering" — covers prompt structure, in-context learning mechanics, and failure modes in more depth than an API quickstart.

Why the source material calls this out specifically: "you cannot improve what you don't measure" — the source curriculum describes evals as the single most under-taught practitioner skill, on the grounds that a prompt change without an eval is just vibes.

Covers: golden sets, exact-match vs. LLM-as-judge scoring, regression-testing prompts.

Huyen deepening: Chapter 3 ("Evaluation Methodology") and Chapter 4 ("Evaluate AI Systems") — the fuller treatment of exact-match vs. AI-judge tradeoffs, golden-set construction, and safety/toxicity evaluation that a single practice day can only introduce.

Covers: embeddings, cosine similarity, chunking strategy, retrieve-then-generate, and why RAG beats simply stuffing more text into the context window.

Huyen deepening: Chapter 6, "RAG and Agents" — the fuller treatment of retrieval strategy design and where RAG's context-construction choices actually fail in production.

Choosing a vector store — by architecture, not by name

The Day 31 exercise doesn't require a specific product — any of these will index 20 documents fine. The choice starts to matter once you're past the toy exercise. Categorize by what you actually need, and confirm current benchmarks/pricing yourself before committing, since this market moves fast and any specific numbers here would be stale within a year.

Covers: when to fine-tune vs. prompt vs. RAG; full fine-tuning vs. parameter-efficient methods; how LoRA's low-rank update matrices work, and why the resulting checkpoint is megabytes rather than gigabytes.

Huyen deepening: Chapter 7, "Finetuning" — the fuller argument on the model-memory arithmetic behind the fine-tune/prompt/RAG decision, plus Chapter 8, "Dataset Engineering," on why a small, carefully-curated instruction set can outperform a much larger noisy one for fine-tuning specifically.

Covers: function calling, the agent loop (reason, act, observe, repeat), stopping conditions — deliberately without an agent framework, so the raw loop is visible.

Huyen deepening: Chapter 6, "RAG and Agents" (the agentic half) — covers evaluating agentic systems specifically, beyond single-turn tool selection.

Past a single agent: multi-agent orchestration

The Day 33 exercise builds one agent with two tools — the right place to start, because you can see the whole reason/act/observe loop directly. Real systems often split responsibility across several agents once a single agent's context and tool set get too broad to reason about reliably. This is a genuine extension of what you just built, not a different technology:

The framing worth keeping: an agent that works on the happy path is a demo; one that fails safely is a product.

Covers: prompt injection, tool-permission boundaries, max-iteration guards, structured tool schemas, trace logging, graceful degradation.

No new material — ship. Strong project options in the source curriculum: a RAG assistant over technical documentation (with evals), an agent that triages a real class of failures, or the tiny GPT from Track 4 extended with one experiment of your own design.

Huyen deepening: Chapter 10, "AI Engineering Architecture and User Feedback" — how the individual techniques from this whole track (prompting, evals, RAG, fine-tuning, agents) assemble into one production-grade system, and how user feedback closes the loop after ship.

Covers: CPU vs. GPU design philosophy — few clever cores vs. thousands of simple ones; streaming multiprocessors, warps, latency hiding; why memory bandwidth, not FLOPs, bounds most real workloads.

Covers: kernels, threads, blocks, grids; threadIdx/blockIdx index math; host vs. device memory; cudaMalloc/cudaMemcpy.

Covers: registers, shared memory, L2 cache, HBM — their relative sizes, latencies, and bandwidths; memory coalescing — why adjacent threads should read adjacent addresses.

Why it echoes later: the same idea — keep hot data in fast memory — is what FlashAttention does at production scale further into this track.

Covers: loading tiles of two matrices into shared memory, synchronizing, and accumulating; why this reduces HBM traffic substantially.

Covers: compute-bound vs. memory-bound vs. overhead-bound workloads; the roofline model; PyTorch profiler basics.

Covers: block-level GPU programming in Triton that compiles to fast device code without raw CUDA; the official vector-add and fused-softmax tutorials.

No new material — consolidate the week into a single comparison.

Covers: fp32 vs. fp16/bf16, tensor cores, loss scaling, mixed-precision training via autocast — why half precision roughly doubles effective bandwidth and unlocks tensor-core throughput at the same time.

Covers: graph capture and codegen under the hood (why fusion eliminates memory-bound overhead), and the one-time compile cold-start cost.

Covers: why naive attention is memory-bound (it materializes the full N×N attention matrix in HBM) and how FlashAttention tiles the computation through fast on-chip memory instead — the Day 39 tiling trick, applied at production scale.

Covers: custom autograd Functions (you write forward and backward yourself), and integrating CUDA/Triton kernels into PyTorch directly.

Karpathy's llm.c: GPT-2 training in raw C/CUDA. The fp32 CPU reference implementation is the bridge from Python-level understanding to systems code.

Covers: int8/4-bit quantization (weight-only vs. dynamic), the KV cache, batching — and why inference is fundamentally a memory-bandwidth game, not a compute one.

No new material — apply the entire toolkit at once.

Covers: nodes, GPUs per node, NVLink within a node, InfiniBand/Ethernet between nodes; the job scheduler as the front door to every HPC system.

Why it matters one day later: the same all-reduce collective operation you implement by hand here is exactly what synchronizes gradients in DDP.

Covers: ranks, communicators, point-to-point send/receive, collectives (broadcast, reduce, all-reduce).

Covers: one process per GPU, a replicated model, gradient all-reduce overlapped with the backward pass.

Covers: why DDP hits a memory wall (weights + gradients + optimizer states add up to roughly 16 bytes per parameter under mixed precision); ZeRO's idea — shard those states across GPUs instead of replicating them; FSDP as PyTorch's native implementation of that idea.

Covers: when a model doesn't fit even after sharding — splitting individual layers across GPUs (tensor parallel) or splitting the layer stack into stages with micro-batching (pipeline parallel); how production systems compose data, tensor, and pipeline parallelism together.

Covers: checkpoint/resume (a large enough job will lose nodes eventually), straggler effects, NCCL as the GPU collective-communication layer, and model FLOPs utilization (MFU) — described in the source material as the one honest metric of training efficiency.

Ship the program. Two strong options from the source curriculum: train your GPT with DDP on multiple GPUs with mixed precision + compile + fused attention + checkpointing, reporting tokens/sec, scaling efficiency, and MFU — or an end-to-end ML service on real infrastructure data with GPU-accelerated training, an eval suite, and a deployment script.