Inference

What Is Inference

Inference is the act of using a model after training. You hand it an input — text, image, audio — and it produces output by running the input through its frozen weights. Training builds the model; inference uses it. Every ChatGPT message, every image generation, every embedding lookup is an inference call. Use the term whenever you’re talking about cost, latency, or scaling of a deployed model rather than the training run.

Training vs Inference

• Training — adjusts the model’s weights using gradient descent over millions of examples. Done once, takes weeks, runs on huge GPU clusters
• Inference — keeps weights frozen, computes a forward pass for one input. Done billions of times, takes milliseconds to seconds, runs on anything from a phone to a multi-GPU server

What Drives Inference Cost

• Model size — a 70B model needs more memory and compute per token than a 7B
• Sequence length — longer prompts and longer outputs each scale roughly linearly
• Batch size — bigger batches use the GPU more efficiently but raise latency for any single user
• Precision — FP16, INT8, INT4 quantization trades accuracy for speed
• Hardware — H100, A100, TPU v5, Apple Silicon, or CPU — each has its own cost-per-token

Inference Stacks

• vLLM — high-throughput open-source server, PagedAttention
• TensorRT-LLM — NVIDIA’s optimized stack
• llama.cpp — CPU and consumer-GPU inference
• Ollama — local model runner built on llama.cpp
• Hosted APIs — OpenAI, Anthropic, Together, Groq, Fireworks abstract the stack away

When someone says “inference is the bottleneck,” they mean it’s the recurring cost — training is one-time, but every user message pays for inference.