AI Inference

What Is AI Inference?

AI inference is running a trained model to produce output: the computation that happens every time an application sends a prompt and gets a response. Training builds the model once; inference is the recurring workload after that, and it is what API pricing, latency budgets, and GPU serving infrastructure all revolve around. That workload is growing at a startling pace: Google reported its monthly inference volume grew 50x in one year, from 9.7 trillion tokens a month to over 480 trillion by May 2025 [1].

Key Takeaways

• Training is a one-time capital cost paid by the model provider. Inference is the operating cost you pay on every call, metered per LLM token.
• Inference has two phases with different economics: prefill, which processes your whole prompt in parallel, and decode, which generates output one token at a time.
• Latency splits the same way. Time to first token reflects prefill; tokens per second reflects decode. Agents that stream long outputs live and die by the second number.
• You rarely need your own GPUs. API providers, serverless inference platforms, and self-hosted open-weight models form a spectrum of cost, control, and operational burden.

How It Works

When a request arrives, the serving stack tokenizes the prompt and runs prefill: every prompt token flows through the transformer model at once, which is highly parallel and fast per token even for huge prompts. Prefill produces the internal state needed to start generating, stored in the KV cache. Then decode begins, appending one token at a time, with each new token depending on everything before it. Decode is sequential and memory-bound, which is why generation speed is measured in tens to low hundreds of tokens per second while prefill chews through thousands.

Serving systems layer optimizations on top. Continuous batching interleaves many users' requests on the same GPUs. Prompt caching reuses prefill work when requests share a prefix, which is why a stable system prompt is cheaper than a shifting one. Quantization shrinks model weights to lower precision so bigger models fit on smaller hardware. Speculative decoding drafts several tokens with a small model and verifies them with the large one, buying speed without quality loss.

For engineers building on agents, inference characteristics shape design. An agent loop is a chain of inference calls where each step waits for the last, so decode speed compounds across steps into wall-clock time a user actually feels. Cost compounds the same way, because each step resends the growing transcript as input. Choosing when a task deserves a frontier model versus a small fast one is fundamentally an inference-economics decision.

Example

A team's coding agent takes four minutes per task, and users complain. Profiling the run shows twelve sequential model calls, with 80 percent of wall-clock time spent in decode on a frontier model that streams around 50 tokens per second. Two changes fix it. Verbose tool outputs get summarized by a small model before entering the transcript, shrinking both prefill cost and the text the big model tends to echo back. And the final step, formatting a report from an already-decided answer, moves to a fast small model entirely. Tasks drop under ninety seconds and per-task inference cost falls by roughly 40 percent, with no change to the diffs the agent produces.

What People Get Wrong

The common mistake is assuming output quality is fixed per model and only speed varies between providers. In practice, the same open-weight model served by different inference stacks can behave differently. Aggressive quantization, altered sampling defaults, shortened context handling, and batching trade-offs all shift accuracy in ways a demo will not reveal. When you swap inference providers, rerun your LLM evals as if you had swapped models, because effectively you have.

FAQ

What is the difference between training and inference? Training adjusts a model's weights by processing huge datasets over weeks on large GPU clusters. Inference freezes those weights and uses them to answer requests. As a builder you almost never train; you consume inference and occasionally fine-tune.

Why is inference expensive if the model is already trained? Every request runs billions of weight multiplications across expensive accelerators, and generation repeats that work for each output token. At product scale, inference spend usually dwarfs what any single customer's share of training would have been. The consolation is that prices fall fast: Stanford's 2025 AI Index found the inference cost of a GPT-3.5-level system dropped more than 280-fold between November 2022 and October 2024 [2].

Should I self-host inference? Only with a clear reason: data residency, latency floors, heavy sustained volume, or a fine-tuned open-weight model. Below serious scale, per-token APIs are cheaper than idle GPUs plus the engineers to run them.

Sources

1. Google. "I/O 2025 keynote: monthly tokens processed grew from 9.7 trillion to over 480 trillion in a year." https://blog.google/technology/ai/io-2025-keynote/. Accessed August 2026.
2. Stanford HAI. "AI Index Report 2025: GPT-3.5-level inference cost fell over 280-fold, November 2022 to October 2024." https://hai.stanford.edu/ai-index/2025-ai-index-report. Accessed August 2026.