LLMs & AI systems
Transformer internals, attention, training and fine-tuning, sampling, inference serving and quantization, prompting and context.
Questions I use to pressure-test my understanding of language models below the API surface — what the architecture is actually doing, how models are trained and adapted, why decoding parameters behave the way they do, what inference costs and how to shrink it, and what degrades over a long conversation.
The agent stack built on top of the model — tool use, MCP, skills, harnesses, multi-agent systems and agent evaluation — lives in the AI agents topic; retrieval and RAG live in Retrieval, search & RAG.
Subtopics
- Model architecture: vectors, encoders, decodersSeniorEmbeddings, tokenization, and the difference between encoder, encoder-decoder, and decoder-only transformers — and why decoder-only won.12 concepts
- Self-attention and positional structureSeniorAttention built from the ground up: what goes in, where Q/K/V and every weight matrix come from, softmax and masking, a worked numeric example, scaling, multi-head shapes, a full PyTorch module, O(n²) cost and FlashAttention, positional encodings (sinusoidal, RoPE, ALiBi), GQA, and what heads learn.13 concepts
- Mixture of experts and architecture familiesSeniorWhy a 120B model can run on ~6B active parameters, how attention and layer design change memory, Qwen vs Llama vs Mistral, and the n-gram lookup table in Qwen3.8-Flash-Next.5 concepts
- Multimodal models: images, audio, videoSeniorHow non-text input becomes tokens, with worked patch and token arithmetic, projectors, fusion architectures, image/audio/video cost and KV-cache numbers, OCR vs VLM, and where multimodal grounding breaks.8 concepts
- Training and post-trainingSeniorPretraining vs SFT vs RLHF/DPO, reward models, the KL penalty, constitutional/RLAIF methods, and why alignment narrows the output distribution.11 concepts
- Fine-tuning mechanics in practiceSeniorFull fine-tune vs LoRA/QLoRA vs prompt-tuning, LoRA maths and PEFT/TRL code, dataset quality and size, regression evals, and when weights beat prompts, RAG or tools.13 concepts
- Decoding parameters: temperature, top-p, top-kSeniorThe sampling pipeline with equations — temperature-scaled softmax, top-k, nucleus sampling, min-p, penalties — how they interact, and what determinism really costs.10 concepts
- Structured output and constrained decodingSeniorHow grammar-constrained decoding guarantees valid syntax, JSON mode vs full CFGs vs tool calling, how a model actually emits a tool call, why it doesn't guarantee correctness, and defensive parsing.10 concepts
- Reasoning models and test-time computeSeniorWhat 'thinking' tokens actually are, how reasoning models are trained, reasoning-effort budgets, process vs outcome reward, and when more thinking hurts.10 concepts
- KV cache, prompt caching, and inference economicsSeniorPrefill vs decode, KV-cache memory math, provider prompt caching and its prefix rule, batching, speculative decoding, and routing between large and small models.11 concepts
- Quantization and inference efficiencySeniorWhat quantization does mathematically, PTQ vs QAT, GPTQ/AWQ/GGUF, fp8 vs int4, a map of formats and the tools that run them, where accuracy loss shows up, and distillation.11 concepts
- VRAM and running LLMs locallyMid / SeniorWhy VRAM is the scarce resource, the weights + KV cache + activations budget, how to run a model on your own machine, and vLLM vs Ollama vs llama.cpp and its forks.8 concepts
- Messages, roles, and prompt structureSeniorWhat system / user / assistant roles actually are, chat templates, prefill, few-shot, where instructions carry the most weight, and why the system prompt cannot keep secrets.9 concepts
- Context, long conversations, and attention decaySeniorWhat 'context' actually is, the context window, lost-in-the-middle, whether attention degrades as history grows, and context-rot and token-budget mitigations.9 concepts
- Evaluating LLMs: benchmarks, contamination, judgesSeniorWhy public leaderboards mislead, benchmark contamination, pass@k, perplexity, confidence intervals on accuracy, Elo-style preference ranking, LLM judges and their agreement statistics, versioning prompts like code, and building an eval you can actually trust.11 concepts
- Safety, prompt injection, and failure modesSeniorWhy instructions and data share one channel, direct vs indirect prompt injection with concrete payloads, exfiltration, jailbreak taxonomy, sycophancy probes, hallucination types, and defense-in-depth in code.9 concepts
- Common misconceptions and gotchasSeniorWhere a surface understanding of determinism, tokenization, context, embeddings, and 'reasoning' diverges from a real one: for each, the wrong belief, why it feels true, the mechanism, and a concrete demonstration.11 concepts