How the field got here

The Transformer

A sequence architecture built entirely on attention, discarding recurrence and convolution.

Why it mattered. Every frontier model since is a descendant. Parallelizable training made web-scale pretraining practical.

Vaswani et al., “Attention Is All You Need” (2017)

Generative pretraining

Pretrain a large model on unlabeled text, then adapt it to tasks — GPT by autoregression, BERT by masked prediction.

Why it mattered. Replaced task-specific architectures with one recipe: pretrain once, specialize cheaply.

Radford et al., GPT (2018); Devlin et al., BERT (2018)

Neural scaling laws

Loss falls as a smooth power law in parameters, data, and compute; Chinchilla (2022) corrected the parameter–data ratio.

Why it mattered. Turned model building into an investment calculus — labs could forecast returns on compute before spending it.

Kaplan et al. (2020); Hoffmann et al., Chinchilla (2022)

In-context learning

GPT-3 showed a large enough model performs new tasks from examples in the prompt, with no weight updates.

Why it mattered. Made prompting the dominant interface to AI and revealed that scale produces qualitatively new behavior.

Brown et al., “Language Models are Few-Shot Learners” (2020)

Retrieval-augmented generation

Ground a model’s output in documents fetched at inference time rather than in weights alone.

Why it mattered. Became the standard bridge between static models and private or fresh data; the backbone of enterprise deployment.

Lewis et al., RAG (2020)

Denoising diffusion models

Generate images by learning to reverse gradual noising; latent diffusion (2022) made it cheap enough for consumer hardware.

Why it mattered. Displaced GANs and powered DALL·E 2, Stable Diffusion, and Midjourney — the image-generation boom of 2022.

Ho et al., DDPM (2020); Rombach et al., Latent Diffusion (2022)

Sparse mixture of experts

Route each token through a small subset of expert subnetworks, decoupling total parameters from per-token compute.

Why it mattered. Now the default frontier architecture — Mixtral, DeepSeek-V3, Llama 4, and (reportedly) GPT-4 are MoE.

Fedus et al., Switch Transformer (2021)

Low-rank adaptation (LoRA)

Fine-tune by learning small low-rank weight deltas instead of updating the full model.

Why it mattered. Cut fine-tuning costs by orders of magnitude and enabled the hobbyist fine-tune ecosystem around open models.

Hu et al., LoRA (2021)

RLHF and instruction tuning

Fine-tune on human demonstrations, then optimize against a learned model of human preferences.

Why it mattered. Turned raw text predictors into usable assistants; the technique behind ChatGPT.

Ouyang et al., InstructGPT (2022)

Chain-of-thought prompting

Ask a model to reason step by step before answering.

Why it mattered. Unlocked multi-step reasoning at inference time and seeded the later reasoning-model paradigm.

Wei et al. (2022)

Constitutional AI

Align a model against a written set of principles using AI-generated feedback (RLAIF) instead of human labels alone.

Why it mattered. Showed alignment training could scale without proportional human labeling; core to Claude’s training.

Bai et al., Constitutional AI (2022)

FlashAttention

An IO-aware exact attention kernel that avoids materializing the full attention matrix.

Why it mattered. Made long contexts computationally tractable; standard in every serious training and inference stack.

Dao et al. (2022)

Weight quantization

Compress trained weights to 8, 4, or fewer bits with minimal quality loss (GPTQ, AWQ, GGUF).

Why it mattered. Let frontier-class open models run on single GPUs and laptops; the foundation of local inference.

Frantar et al., GPTQ (2022); Lin et al., AWQ (2023)

Efficient inference serving

PagedAttention (vLLM) manages KV-cache like virtual memory; continuous batching keeps GPUs saturated; speculative decoding drafts tokens with a small model.

Why it mattered. Cut serving costs several-fold and made high-throughput API serving economically viable.

Kwon et al., vLLM (2023); Leviathan et al. (2023)

Context-length extension

Positional-encoding interpolation and scaling (RoPE variants, YaRN) stretched contexts from 4K to 128K+ tokens.

Why it mattered. Whole-codebase and whole-document reasoning became routine; long context now substitutes for fine-tuning in many uses.

Chen et al. (2023); Peng et al., YaRN (2023)

Selective state-space models

Mamba matched transformer quality at small scale with linear-time sequence mixing.

Why it mattered. The strongest challenge yet to attention’s monopoly; hybrid SSM–attention layers now appear in production models.

Gu & Dao, Mamba (2023)

Native multimodality

Single models trained end-to-end across text, images, and audio (GPT-4o, Gemini), rather than stitched-together encoders.

Why it mattered. Real-time voice conversation and unified any-to-any interfaces; multimodal became the frontier default.

Test-time compute scaling

Train models by reinforcement learning to produce long private chains of thought, spending more inference compute on harder problems (o1, R1).

Why it mattered. Opened a second scaling axis beyond pretraining; reasoning models now dominate math, science, and coding benchmarks.

OpenAI o1 (2024); DeepSeek-R1 (2025)

Computer use and agentic tool use

Models operate real interfaces — terminals, browsers, GUIs — in a perception–action loop.

Why it mattered. Moved AI from answering questions to performing work; the basis of coding agents and the emerging agent economy.

Anthropic computer use (2024)

Model Context Protocol

An open standard connecting models to tools and data sources, adopted across major labs and editors in 2025.

Why it mattered. Did for agent tooling what USB did for peripherals: one integration, every model.

Anthropic, MCP (2024)

Reasoning distillation

DeepSeek-R1 showed reasoning behavior can be distilled from RL-trained teachers into small open models, and that pure RL can elicit it without supervised scaffolding.

Why it mattered. Collapsed the cost of reasoning capability and spread it through the open ecosystem within months.

DeepSeek-R1 (2025)

Verifiable rewards and RL environments

Post-training shifted from learned preference models to rewards a program can check — test suites, math answers, proof checkers — with GRPO as the workhorse algorithm.

Why it mattered. Made capability a function of environment quality; building rich RL environments and verifiers became the field’s new bottleneck and an industry of its own.

Shao et al., GRPO (2024); Lambert et al., Tülu 3 (2024)

Diffusion language models

Generate text by parallel iterative denoising instead of one token at a time; Mercury and Gemini Diffusion demonstrated 1,000+ tokens per second at competitive quality.

Why it mattered. The first credible break with autoregression for text — an order-of-magnitude latency advantage that matters for coding assistants and agent loops.

Nie et al., LLaDA (2025); Inception Labs, Mercury (2025)

Circuit tracing and attribution graphs

Cross-layer transcoders turn a forward pass into a readable graph of interacting features, applied to a production model (Claude 3.5 Haiku) and open-sourced.

Why it mattered. Interpretability graduated from toy models to deployed ones — exposing planning ahead, shared multilingual circuits, and unfaithful reasoning; named a 2026 MIT Technology Review breakthrough.

Lindsey et al., “On the Biology of a Large Language Model” (2025)

Video generation and world models

Veo 3 and Sora 2 generated physics-plausible video with synchronized audio; Genie 3 produced explorable 720p worlds in real time that stay coherent for minutes.

Why it mattered. Video crossed from novelty to production tool, and promptable interactive worlds opened a path to training agents in simulation.

Google DeepMind, Veo 3 and Genie 3 (2025); OpenAI, Sora 2 (2025)

Muon at frontier scale

A matrix-orthogonalizing optimizer — stabilized by MuonClip’s qk-clip — trained the trillion-parameter Kimi K2 on 15.5T tokens with zero loss spikes.

Why it mattered. The first optimizer to displace Adam in frontier training, with roughly 2× token efficiency; optimizer research reopened after a decade of default AdamW.

Jordan et al., Muon (2024); Kimi Team, Kimi K2 (2025)

Olympiad-gold mathematical reasoning

General-purpose reasoning models from OpenAI and Google DeepMind solved 5 of 6 IMO problems in natural language under contest conditions, reaching the gold-medal bar.

Why it mattered. Frontier reasoning crossed a threshold humans use to certify their best; the techniques — scaled RL and parallel test-time search — fed back into production models within months.

OpenAI experimental reasoning model; Google DeepMind, Gemini Deep Think (2025)

Hybrid linear attention in production

Frontier open models (Qwen3-Next, Kimi Linear) interleave gated linear-attention layers with full attention at roughly 3:1, keeping KV-cache costs near-constant with length.

Why it mattered. The state-space promise cashed in at scale — long-context serving stopped scaling quadratically, and pure softmax attention lost its monopoly on frontier architecture.

Yang et al., Gated DeltaNet (2024); Kimi Team, Kimi Linear (2025)

Natively trainable sparse attention

DeepSeek trained attention to be sparse from the start — a lightweight indexer selects the few tokens worth full attention (NSA, then DSA in DeepSeek-V3.2).

Why it mattered. Cut long-context compute from quadratic to near-linear with negligible quality loss, roughly halving long-context API costs.

Yuan et al., Native Sparse Attention (2025); DeepSeek-V3.2 (2025)

Long-horizon agents

Context management, memory, and orchestrator–worker sub-agent architectures let models sustain coherent work over hours and thousands of tool calls.

Why it mattered. The unit of delegation shifted from the prompt to the task; autonomous coding sessions became products rather than demos.

Anthropic, “How we built our multi-agent research system” (2025)

Agent memory and learning from experience

Agents distill their own trajectories into reusable strategies — workflow memories, reasoning banks, RL-trained memory managers — improving without weight updates.

Why it mattered. A practical answer to continual learning: the limiting factor for deployed agents shifted from raw model capability to how experience is stored, consolidated, and reused.

Wang et al., Agent Workflow Memory (2024)

The inference build-out

Rack-scale systems (GB200 NVL72) and wafer-scale chips pushed serving toward 1,000 tokens per second on trillion-parameter models; NVIDIA absorbed Groq’s LPU line as a decode co-processor.

Why it mattered. Roughly 10× better tokens-per-watt generation over generation made long reasoning traces and always-on agents economically routine; inference, not training, now drives the hardware market.