Learn to serve open LLMs with vLLM: KV cache, continuous batching, GPU sizing, Kubernetes GPU nodes, load tests and cost per 1,000 requests.
The second evening of acme-shop's sale. The customer support assistant, an 8 billion parameter open model, runs on one GPU inside a FastAPI service...
vLLM ships as a Python package with a vllm serve command, and as an official Docker image. Both start the same OpenAI-compatible server.
LLM performance has more than one latency, and they pull against throughput. Knowing which number your users feel decides how you tune.
Sizing starts with arithmetic you can do on paper, then a measurement to confirm it.
LLMOps and Production AI Systems covered Deployments, Services and the Horizontal Pod Autoscaler for an AI API.
LLMOps and Production AI Systems covered the managed-versus-self-hosted decision as a whole: data control, operations effort, model choice.
The second evening of acme-shop's sale. The customer support assistant, an 8 billion parameter open model, runs on one...
vLLM ships as a Python package with a vllm serve command, and as an official Docker image.
LLM performance has more than one latency, and they pull against throughput.
Sizing starts with arithmetic you can do on paper, then a measurement to confirm it.
LLMOps and Production AI Systems covered Deployments, Services and the Horizontal Pod Autoscaler for an AI API.
LLMOps and Production AI Systems covered the managed-versus-self-hosted decision as a whole: data control, operations...
This topic is optional. It names the techniques you will meet when models outgrow one GPU or when latency targets get...
This lab serves a small open model with vLLM, checks its KV cache arithmetic against your own estimate, load tests it...
Quick reference Common mistakes Serving an LLM with a plain generate loop in a web framework is how the acme-shop...
Roles that use the skills in this module.
AI Engineer
MLOps Engineer
Platform Engineer
DevOps Engineer
Not a software engineer sheet. Every problem comes from real DevOps, SRE, Platform and Cloud interviews, from your first script to a system you build yourself.
Open the Coding SheetA basic Transformers service often processes requests one at a time or in fixed batches, which leaves GPU capacity unused when requests have different prompt and reply lengths. vLLM uses continuous batching to schedule new requests as others finish, and PagedAttention to manage KV cache memory in small blocks. Under concurrent load this usually gives much higher throughput and better memory use on the same GPU, though the exact gain depends on the model and traffic.
vLLM has a CPU backend with its own images and installation steps, and it works for small models and learning. It is much slower than a GPU, so treat CPU numbers as functional tests, not capacity measurements. For this module's lab, a free notebook GPU such as a T4 gives realistic results when one is available.
Start with the weights: parameters times bytes per parameter, so an 8 billion parameter model needs about 16 GB in 16-bit precision. Then add the KV cache, which grows with context length and the number of concurrent requests, plus some working memory. The sizing topic in this module shows the formula and a worked example.
Self-hosting is cheaper only when your GPUs stay busy. You pay for each GPU every hour whether it serves one request or a thousand, while a hosted API charges per token. Measure throughput at your latency targets with realistic prompts, estimate how much of that capacity real traffic uses, count every replica you run, and compare cost per 1,000 requests; at low or spiky traffic, hosted APIs usually win.