Serve an LLM with vLLM and Monitor Cost and Quality

Serve an open LLM with vLLM on CPU or GPU, monitor TTFT, queueing and KV cache, sample live answer quality, and estimate compute cost per request.

Architecture Overview & Problem Statement

This project serves a small open model, Qwen2.5-0.5B-Instruct, with vLLM's OpenAI-compatible server, CPU first and GPU optional. It then operates the model the way a platform team would:

Explicit memory sizing: the container gets a hard memory limit.
Server metrics: Prometheus watches time to first token (TTFT), queueing and KV cache pressure.
Load testing: concurrency is swept to find where latency breaks.
Cost: a capacity-based compute cost estimate per 1,000 requests.
Live quality: a quality signal is sampled from real traces, grouped by prompt version and model.

The support assistant, MLflow tracing and golden-set gate come from "LLM Observability and Prompt Versioning with MLflow". This project reuses a compact version of them and adds what serving needs: redaction before tracing, and a CI gate whose harness the pull request cannot change.

The problem

acme-shop wants to stop paying a hosted API for its support assistant and run an open model itself. Three questions decide whether that works:

How many requests can one machine handle before users wait?
What does a request cost at that load?
How do we notice, in production, that a prompt change made the assistant invent policies?

This project answers each one with a measurement, not a guess.

What you build

flowchart TD U([Customer question]) --> R[redact + retrieve
assistant.py] R --> V[vLLM
OpenAI API :8000] R -.->|redacted spans| M[(MLflow traces)] V -->|/metrics| P[Prometheus rules] M --> Q[sample_review.py
by prompt version] B[bench.py sweep] --> V classDef app fill:#DBEAFE,color:#1E293B,stroke:#326CE5 classDef serve fill:#10B981,color:#FFFFFF,stroke:#10B981 classDef obs fill:#EDE9FE,color:#1E293B,stroke:#8B5CF6 class U,R,B app class V serve class M,P,Q obs

How a change is judged in CI

sequenceDiagram participant PR as Pull request participant CI as Job (no secrets) participant V as vLLM container CI->>CI: check out base SHA (harness)
and PR (candidate) CI->>CI: published prompts unchanged? CI->>V: base app + base prompt CI->>V: PR app + PR prompt CI-->>PR: base gate decides

Repository layout

◈ DIAGRAM

acme-support-llm/
├── compose.yaml  compose.gpu.yaml   # vLLM (CPU default, GPU override) + Prometheus
├── monitoring/                      # prometheus.yml, alerts.yml, check_metrics.sh
├── bench/bench.py                   # concurrency sweep, TTFT, compute cost estimate
├── app/assistant.py  app/redact.py  # reused assistant + redaction before tracing
├── prompts/  knowledge/  golden/    # prompt versions (immutable), policies, 16 golden cases
├── eval/run_golden.py  scoring.py  gate.py   # the harness CI takes from the base commit
├── quality/                         # simulate_live.py, sample_review.py, retention.py
├── tests/                           # test_gate.py, fake_llm.py (offline stand-in)
└── .github/workflows/llm-eval.yml

Prerequisites (by module title)

Serving LLMs with vLLM and GPU Inference (required: this project assumes its concepts).
LLM Observability and Prompt Versioning with MLflow (the assistant, tracing and golden-set gate are reused from it).
Monitoring and Logging: Prometheus to Tracing, AI Evaluation and Testing.
Tools: Docker with Compose, at least 8 GB of memory for Docker's VM (a minimum to try cautiously, not a guarantee), about 3 GB of disk, Python 3.12.

Versions used

Tool	Version	Checked
vLLM (CPU / GPU images)	v0.33.0: `vllm-openai-cpu:v0.33.0-x86_64` / `vllm-openai:v0.33.0`	GitHub tag and vLLM CPU docs, 5 Nov 2026
MLflow / openai SDK	3.18.0 / 3.46.0	PyPI, 5 Nov 2026
Prometheus	v3.15.1	GitHub tag, 5 Nov 2026
Model	Qwen/Qwen2.5-0.5B-Instruct	Hugging Face (not downloaded here)

Security
vLLM itself was NOT run for this guide. The sandbox cannot download model weights from Hugging Face or pull Docker images. Everything else ran against tests/fake_llm.py, an OpenAI-compatible stand-in that is not a language model, shown in full in Milestone 2. The stand-in ran:

redaction, verified on stored traces

MLflow tracing

the golden set and gate, including the base versus candidate harness

the immutable-prompt check

live quality sampling and trace retention

the concurrency sweep and cost arithmetic

the 7 tests

Prometheus rules passed promtool, which checks syntax only. Metric names were read from vLLM v0.33.0's source code, not from a running server. The CI workflow was checked for YAML validity only. Until you run the CPU deployment yourself and record your machine, this is not a verified vLLM serving project. Every stand-in number below says so.

Milestone 1: Serve Qwen2.5-0.5B with vLLM on CPU

This milestone starts vLLM's OpenAI-compatible server on CPU with a hard container memory limit, and explains where that memory goes.

Where the memory goes

Consumer	Rough size here	Set by
Model weights (0.5B parameters, bf16)	about 1 GB	the model and `--dtype`
KV cache	2 GiB	`--kv-cache-memory-bytes`
Python, PyTorch, vLLM runtime, `/dev/shm`	1 to 3 GB	the image; `shm_size`

mem_limit: 6g caps the whole container. shm_size: 2g is not a cap: it sizes the shared-memory filesystem, which is RAM and counts toward the limit. If the container is killed with exit code 137, it hit mem_limit. Lower the KV cache or --max-model-len before raising the limit. These sizes are estimates for planning; watch docker stats on your machine.

The KV cache is where concurrency comes from. Every running request holds cache blocks for its tokens, so a larger cache or a smaller --max-model-len means more requests in flight. --kv-cache-memory-bytes sets it on CPU; vLLM's CPU docs call the older VLLM_CPU_KVCACHE_SPACE variable legacy. On GPU, --gpu-memory-utilization plays that role.

Step 1: Write the Compose files

compose.yaml:

YAML

# CPU path (default, free): vLLM's CPU image + Prometheus. First start downloads ~1 GB of weights.
# Needs Docker's VM to have more than mem_limit (8 GB is a minimum to try, not a guarantee).
services:
  vllm:
    image: vllm/vllm-openai-cpu:v0.33.0-x86_64      # arm64 hosts: v0.33.0-arm64
    command:
      - --model=Qwen/Qwen2.5-0.5B-Instruct
      - --served-model-name=acme-support
      - --max-model-len=4096
      - --kv-cache-memory-bytes=2147483648          # 2 GiB of KV cache; size it to your RAM
      - --dtype=bfloat16                            # if your CPU lacks bf16 support, try float32
    shm_size: 2g        # /dev/shm for worker communication; it is RAM and counts toward mem_limit
    mem_limit: 6g       # hard cap for the whole container: weights + KV cache + runtime + shm
    ports: ["8000:8000"]
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface   # keep the weights between runs
  prometheus:
    image: prom/prometheus:v3.15.1
    command: ["--config.file=/etc/prometheus/prometheus.yml"]
    volumes:
      - ./monitoring/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./monitoring/alerts.yml:/etc/prometheus/alerts.yml:ro
    ports: ["9090:9090"]

compose.gpu.yaml (optional; costs money on a cloud GPU; the memory limit does not apply to GPU memory):

YAML

# OPTIONAL GPU path (costs money on a cloud GPU). Use: docker compose -f compose.yaml -f compose.gpu.yaml up
services:
  vllm:
    image: vllm/vllm-openai:v0.33.0
    command:
      - --model=Qwen/Qwen2.5-0.5B-Instruct
      - --served-model-name=acme-support
      - --max-model-len=4096
      - --gpu-memory-utilization=0.85               # share of GPU memory for weights + KV cache
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

Step 2: Start and check

Bash

docker compose up -d vllm
docker compose logs -f vllm            # wait for "Application startup complete"
curl -s localhost:8000/v1/models
curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"acme-support","messages":[{"role":"user","content":"Say hello in five words."}],"max_tokens":20}'
docker stats --no-stream                # memory in use vs the 6 GiB limit

Expected output shape (not run for this guide): /v1/models lists acme-support, and the chat call returns choices[0].message.content with a usage block. Write down CPU model, core count, RAM and docker stats memory: every later number depends on them.

Acceptance check

docker stats shows the container under its limit after a few requests, and your machine spec is in the README.

Milestone 2: Connect the Assistant, Redacted and Traced

This milestone points the assistant from the MLflow module at vLLM, adds redaction before anything is traced or sent to the model, and verifies the stored traces contain no personal data.

Step 1: Reuse the assistant and add redaction

app/redact.py drops emails, order ids, Indian mobile numbers and card numbers. The policy assistant never needs them, so they are removed, not masked later:

Python

"""Remove personal data from a customer message BEFORE it reaches the model or a trace.
 
The FAQ assistant never needs an email, phone number or order id to answer a policy
question, so we drop them (data minimisation) instead of masking them later.
"""
import re
 
PATTERNS = [
    ("EMAIL", re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+")),
    ("ORDER", re.compile(r"\bACME-\d{6,}\b", re.IGNORECASE)),
    ("PHONE", re.compile(r"(?<!\d)(?:\+91[\s-]?)?[6-9]\d{4}[\s-]?\d{5}(?!\d)")),   # Indian mobile numbers
    ("CARD", re.compile(r"(?<!\d)(?:\d[ -]?){13,16}(?!\d)")),
]
 
 
def redact(text: str) -> str:
    for label, pattern in PATTERNS:
        text = pattern.sub(f"[{label}]", text)
    return text

app/assistant.py (the compact version used here; answer() redacts before the traced function, so neither the spans nor the model ever see raw data):

Python

"""acme-shop support assistant: retrieve policy lines, ask the model, trace everything.
 
Talks to any OpenAI-compatible server (vLLM in this project) via LLM_BASE_URL.
MLflow traces each call: a root span, a retrieval span, and the LLM span from autolog.
This is the minimal version of the assistant from "LLM Observability and Prompt Versioning
with MLflow", plus redaction; reuse your own from that module if you have it.
"""
import os
import re
import time
from pathlib import Path
 
import mlflow
import yaml
from openai import OpenAI
 
from app.redact import redact
 
BASE_URL = os.environ.get("LLM_BASE_URL", "http://127.0.0.1:8000/v1")
MODEL = os.environ.get("LLM_MODEL", "acme-support")          # vLLM --served-model-name
CLIENT = OpenAI(base_url=BASE_URL, api_key=os.environ.get("LLM_API_KEY", "not-needed-locally"))
FAQ = [l.strip() for l in Path("knowledge/faq.md").read_text().splitlines() if l.strip() and not l.startswith("#")]
mlflow.openai.autolog()                                       # traces every chat.completions call
 
 
STOP = {"the", "a", "an", "is", "i", "my", "do", "can", "for", "to", "of", "how", "what", "you", "your"}
 
 
def words(text: str) -> set[str]:
    """Lower-case words without stop words, with a crude plural strip (refunds -> refund)."""
    return {t[:-1] if len(t) > 3 and t.endswith("s") else t for t in re.findall(r"[a-z0-9]+", text.lower())} - STOP
 
 
@mlflow.trace(span_type="RETRIEVER")
def retrieve(question: str, k: int = 3, min_overlap: int = 2) -> list[str]:
    """Keyword overlap is enough for 10 policy lines; swap in a vector store for real corpora.
    min_overlap keeps one shared word like "order" from pulling an unrelated policy into an
    off-topic question: an empty context is what lets the model say it does not know."""
    q = words(question)
    ranked = sorted(FAQ, key=lambda line: len(q & words(line)), reverse=True)
    return [line for line in ranked[:k] if len(q & words(line)) >= min_overlap]
 
 
def load_prompt(version: str) -> dict:
    return yaml.safe_load(Path(f"prompts/{version}.yaml").read_text())
 
 
def answer(question: str, prompt_version: str = "v1") -> dict:
    """Public entry point: redact first, so raw personal data never enters a span or the model."""
    return _answer(redact(question), prompt_version)
 
 
@mlflow.trace(span_type="CHAIN", name="answer")
def _answer(question: str, prompt_version: str) -> dict:
    prompt = load_prompt(prompt_version)
    context = retrieve(question)
    mlflow.update_current_trace(tags={"prompt_version": prompt_version, "model": MODEL})
    start = time.perf_counter()
    resp = CLIENT.chat.completions.create(
        model=MODEL, temperature=0, max_tokens=200,
        messages=[{"role": "system", "content": prompt["system"]},
                  {"role": "user", "content": "Policy context:\n" + "\n".join(f"- {c}" for c in context)
                                              + f"\n\nCustomer question: {question}"}],
    )
    return {"text": resp.choices[0].message.content, "latency_s": time.perf_counter() - start,
            "prompt_tokens": resp.usage.prompt_tokens, "completion_tokens": resp.usage.completion_tokens}

requirements.txt:

TEXT

openai==3.46.0
mlflow==3.18.0
pyyaml==6.0.3
pytest==9.2.0

The prompts, knowledge base and golden set are the module's pattern with this project's content: prompts/v1.yaml, prompts/v2.yaml (a "friendlier" edit that drops the refusal rule), knowledge/faq.md (10 fictional policies) and golden/support.jsonl (12 in-scope, 4 out-of-scope cases).

YAML

version: v1
system: |
  You are acme-shop's support assistant. Answer ONLY from the policy context.
  Quote the exact numbers (days, rupees) from the context.
  If the answer is not in the context, say you don't know and ask the customer to contact support.
  Keep answers under 60 words.

YAML

version: v2
# Candidate change: "friendlier, more helpful answers". It drops the refusal rule.
system: |
  You are acme-shop's friendly support assistant. Be warm and always give the customer a helpful answer.
  Use the policy context and quote the exact numbers (days, rupees).
  Keep answers under 60 words.

Step 2: The offline stand-in (for when you have no model running)

tests/fake_llm.py speaks enough of the OpenAI API to exercise every script without weights. It runs on port 8001 so it never collides with vLLM on 8000:

Python

"""OFFLINE STAND-IN for vLLM's OpenAI-compatible API. Not a language model.
 
It lets you exercise tracing, the golden-set gate and the benchmark maths without weights:
* answers by quoting every context line that shares a word with the question,
  so its score depends on what the app RETRIEVED (a real model would rephrase)
* if nothing overlaps: refuses when the system prompt contains a refusal rule,
  otherwise invents a confident answer (to imitate a hallucination)
* sleeps ~8 ms per output token to imitate decoding
Run: python tests/fake_llm.py --port 8001   (vLLM keeps 8000; point LLM_BASE_URL at 8001)
"""
import json
import re
import time
import uuid
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
 
STOP = {"the", "a", "an", "is", "i", "my", "do", "can", "for", "to", "of", "how", "what", "you", "your"}
 
 
def w(t):
    return {x[:-1] if len(x) > 3 and x.endswith("s") else x for x in re.findall(r"[a-z0-9]+", t.lower())} - STOP
 
 
def reply(messages):
    system = messages[0]["content"].lower()
    user = messages[-1]["content"]
    question = user.split("Customer question:")[-1]
    lines = [l[2:] for l in user.splitlines() if l.startswith("- ")]
    hits = [l for l in lines if w(question) & w(l)]
    if hits:
        return " ".join(hits)
    if "don't know" in system:
        return "I don't know based on our policies. Please contact support."
    return "Yes! We are happy to offer that at no extra cost on every order."
 
 
class H(BaseHTTPRequestHandler):
    def log_message(self, *args):
        pass
 
    def do_GET(self):
        body = b'{"object":"list","data":[{"id":"acme-support","object":"model"}]}'
        self.send_response(200); self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body))); self.end_headers(); self.wfile.write(body)
 
    def do_POST(self):
        req = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
        text = reply(req["messages"])
        toks = text.split()
        usage = {"prompt_tokens": sum(len(m["content"].split()) for m in req["messages"]),
                 "completion_tokens": len(toks), "total_tokens": 0}
        rid, now = f"chatcmpl-{uuid.uuid4().hex[:8]}", int(time.time())
        if req.get("stream"):
            self.send_response(200); self.send_header("Content-Type", "text/event-stream"); self.end_headers()
            time.sleep(0.03)                                   # "prefill"
            for i, t in enumerate(toks):
                time.sleep(0.008)
                chunk = {"id": rid, "object": "chat.completion.chunk", "created": now, "model": req["model"],
                         "choices": [{"index": 0, "delta": {"content": t + (" " if i < len(toks) - 1 else "")},
                                      "finish_reason": None}]}
                self.wfile.write(f"data: {json.dumps(chunk)}\n\n".encode())
            last = {"id": rid, "object": "chat.completion.chunk", "created": now, "model": req["model"],
                    "choices": [], "usage": usage}
            self.wfile.write(f"data: {json.dumps(last)}\n\ndata: [DONE]\n\n".encode())
            return
        time.sleep(0.03 + 0.008 * len(toks))
        body = json.dumps({"id": rid, "object": "chat.completion", "created": now, "model": req["model"],
                           "choices": [{"index": 0, "message": {"role": "assistant", "content": text},
                                        "finish_reason": "stop"}], "usage": usage}).encode()
        self.send_response(200); self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body))); self.end_headers(); self.wfile.write(body)
 
 
if __name__ == "__main__":
    import argparse
    ap = argparse.ArgumentParser()
    ap.add_argument("--port", type=int, default=8001)
    port = ap.parse_args().port
    print(f"stand-in (NOT a model) on http://127.0.0.1:{port}/v1")
    ThreadingHTTPServer(("127.0.0.1", port), H).serve_forever()

Bash

python3.12 -m venv .venv && source .venv/bin/activate && pip install -r requirements.txt
export MLFLOW_TRACKING_URI=sqlite:///$PWD/mlflow.db MLFLOW_DISABLE_TELEMETRY=true
python tests/fake_llm.py --port 8001 &                      # stand-in
export LLM_BASE_URL=http://127.0.0.1:8001/v1                # or http://127.0.0.1:8000/v1 for vLLM

It validates plumbing: tracing, scoring, the gate, the benchmark maths. It says nothing about model quality or serving speed.

Step 3: Send traffic and check the stored traces

quality/simulate_live.py sends 200 requests, 10 percent on prompt v2 as a canary, two questions carrying an email, an order id and a phone number:

Python

"""Send 'production' traffic through the assistant: 90% on prompt v1, 10% canary on v2.
Some questions carry personal data, to prove redaction works on real traces.
"""
import json
import random
import sys
 
import mlflow
 
sys.path.insert(0, ".")
from app.assistant import answer  # noqa: E402
 
random.seed(7)
qs = [json.loads(l)["question"] for l in open("golden/support.jsonl")]
qs += ["My email is priya.k@example.com and order ACME-1234567, when will my refund reach my card?",
       "Call me on +91 98765 43210, can I get cash back for a gift card?"]
mlflow.set_experiment("acme-support-live")
n = int(sys.argv[1]) if len(sys.argv) > 1 else 200
for i in range(n):
    answer(random.choice(qs), "v2" if random.random() < 0.10 else "v1")
mlflow.flush_trace_async_logging()
print(f"sent {n} live requests (about 10% on canary prompt v2)")

Bash

python quality/simulate_live.py 200
python - <<'PY'
import json, mlflow
exp = mlflow.get_experiment_by_name("acme-support-live")
traces = mlflow.search_traces(locations=[exp.experiment_id], max_results=1000, return_type="list")
blob = json.dumps([t.to_dict() for t in traces])
for raw in ["priya.k@example.com", "ACME-1234567", "98765 43210"]:
    print(raw, "FOUND" if raw in blob else "not found")
PY

Expected output (real run, stand-in; the check reads the stored traces, not a report):

TEXT

searched 200 stored traces for raw personal data:
  'priya.k@example.com': not found
  'ACME-1234567': not found
  '98765 43210': not found
  redaction markers present: True

Acceptance check

No raw email, order id or phone number appears in any stored trace, and the [EMAIL], [ORDER], [PHONE] markers do.

Security
Redaction is pattern-based and will miss things (a name, an address). Keep traces behind the same access control as support tickets, and delete them on a schedule: quality/retention.py deletes live traces older than RETENTION_DAYS (default 30). Real run: deleted 0 traces older than 30 days (all traces were minutes old).

Milestone 3: Monitor vLLM with Prometheus

This milestone scrapes vLLM's /metrics, loads alert and recording rules for TTFT, queueing, KV cache and preemptions, and checks the metric names exist on your running server.

Step 1: Configure Prometheus

monitoring/prometheus.yml:

YAML

global:
  scrape_interval: 15s
rule_files:
  - alerts.yml
scrape_configs:
  - job_name: vllm                 # vLLM serves Prometheus metrics on /metrics of the API port
    static_configs:
      - targets: ["vllm:8000"]

monitoring/alerts.yml:

YAML

# Example rules for vLLM v0.33 metric names. Thresholds are illustrative: set them from your own baseline.
groups:
  - name: acme-support-llm
    rules:
      - record: acme:output_tokens_per_hour
        expr: sum(rate(vllm:generation_tokens_total[5m])) * 3600
      - alert: SlowFirstToken
        expr: histogram_quantile(0.95, sum by (le) (rate(vllm:time_to_first_token_seconds_bucket[5m]))) > 2
        for: 10m
        labels: {severity: page}
        annotations:
          summary: "p95 time to first token above 2 s: users stare at a blank chat box."
      - alert: RequestsQueueing
        expr: max(vllm:num_requests_waiting) > 5
        for: 5m
        labels: {severity: ticket}
        annotations:
          summary: "Requests are waiting for a slot: add capacity or cap max_tokens."
      - alert: KvCacheNearlyFull
        expr: max(vllm:kv_cache_usage_perc) > 0.9
        for: 10m
        labels: {severity: ticket}
        annotations:
          summary: "KV cache above 90%: expect preemptions and slower requests."
      - alert: Preemptions
        expr: sum(increase(vllm:num_preemptions_total[10m])) > 0
        labels: {severity: ticket}
        annotations:
          summary: "vLLM preempted requests (KV cache pressure)."

Step 2: Prove the metrics exist on YOUR server

promtool check rules passed (real run: SUCCESS: 5 rules found), but it checks syntax, not whether vllm:kv_cache_usage_perc exists in your image. The names above were read from vLLM v0.33.0's source; verify them at runtime:

monitoring/check_metrics.sh:

Bash

#!/usr/bin/env bash
# promtool only checks rule SYNTAX. This checks that every vLLM metric the rules use
# actually exists on YOUR running server (run after sending at least one request).
set -uo pipefail
URL="${1:-http://127.0.0.1:8000/metrics}"
BODY=$(curl -sf "$URL") || { echo "cannot read $URL"; exit 1; }
MISSING=0
for m in vllm:time_to_first_token_seconds_bucket vllm:num_requests_waiting vllm:kv_cache_usage_perc \
         vllm:num_preemptions_total vllm:generation_tokens_total; do
  if grep -q "^${m}" <<<"$BODY"; then echo "found    $m"; else echo "MISSING  $m"; MISSING=1; fi
done
exit $MISSING

Bash

docker compose up -d prometheus
./monitoring/check_metrics.sh            # after at least one request; exit 1 if any are MISSING

Expected output shape (not run): five found lines. A MISSING line means the metric was renamed in your image: fix alerts.yml before trusting it.

Tip
These rules fire inside Prometheus; nobody is notified. Alert delivery (Alertmanager with a receiver such as email or Slack) is out of scope here and covered in Monitoring and Logging: Prometheus to Tracing.

What each metric tells you

Metric	Meaning	Act when
`vllm:time_to_first_token_seconds`	queue + prefill time	p95 above your chat budget
`vllm:num_requests_waiting`	requests without a slot	above zero for minutes
`vllm:kv_cache_usage_perc`	KV cache in use	near 1.0, with preemptions

Acceptance check

check_metrics.sh exits 0, and Prometheus Status, Targets shows vllm as UP.

Milestone 4: Sweep Concurrency and Estimate Compute Cost

This milestone streams requests at 1, 4, 8 and 16 concurrent users, measures TTFT and throughput at each, and turns throughput into an estimated compute cost per 1,000 successful requests.

What the cost number is, and is not

estimated compute cost per 1,000 successful requests = hourly machine price ÷ (successful requests per second × 3,600) × 1,000

This is a capacity-based estimate of compute only. It assumes the machine stays this busy for the whole hour and leaves out idle time, storage, network, monitoring and operations. At 10 percent utilisation the real cost per request is ten times higher. The benchmark calls the inference endpoint directly, so retrieval, redaction and tracing are not in the timed path.

Step 1: Write the benchmark

bench/bench.py:

Python

"""Load test the inference server at several concurrency levels and estimate compute cost.
 
Streams every request so time to first token (TTFT) is measured, not guessed. It calls the
OpenAI-compatible endpoint directly: retrieval, redaction and tracing are not in the timed path.
 
estimated_compute_cost_per_1000_successful_requests =
    hourly machine price / (successful requests per second x 3600) x 1000
A capacity-based estimate of compute only: it assumes the machine stays this busy all hour and
excludes idle time, storage, network, monitoring and operations. It is not a production bill.
"""
import argparse
import json
import math
import sys
import time
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
 
sys.path.insert(0, ".")
from app.assistant import CLIENT, MODEL, load_prompt, retrieve  # noqa: E402
 
QUESTIONS = [json.loads(l)["question"] for l in open("golden/support.jsonl")]
SYSTEM = load_prompt("v1")["system"]
 
 
def one(i: int) -> dict:
    q = QUESTIONS[i % len(QUESTIONS)]
    msgs = [{"role": "system", "content": SYSTEM},
            {"role": "user", "content": "Policy context:\n" + "\n".join(retrieve(q)) + f"\n\nCustomer question: {q}"}]
    start, first, tokens = time.perf_counter(), None, 0
    try:
        stream = CLIENT.chat.completions.create(model=MODEL, messages=msgs, temperature=0, max_tokens=200,
                                                stream=True, stream_options={"include_usage": True})
        for chunk in stream:
            if chunk.choices and chunk.choices[0].delta.content and first is None:
                first = time.perf_counter()
            if chunk.usage:
                tokens = chunk.usage.completion_tokens
        return {"ok": True, "ttft": (first or time.perf_counter()) - start, "e2e": time.perf_counter() - start,
                "tokens": tokens}
    except Exception as err:   # count failures instead of crashing the benchmark
        return {"ok": False, "error": type(err).__name__, "e2e": time.perf_counter() - start, "tokens": 0}
 
 
def pct(xs, p):
    s = sorted(xs)
    return s[max(0, math.ceil(p * len(s)) - 1)] if s else float("nan")
 
 
def run(n_requests: int, concurrency: int, price: float, label: str) -> dict:
    t0 = time.perf_counter()
    with ThreadPoolExecutor(concurrency) as pool:
        res = list(pool.map(one, range(n_requests)))
    wall = time.perf_counter() - t0
    ok = [r for r in res if r["ok"]]
    rps = len(ok) / wall
    out_tps = sum(r["tokens"] for r in ok) / wall
    return {
        "label": label, "model": MODEL, "requests": n_requests, "concurrency": concurrency,
        "errors": len(res) - len(ok), "wall_s": round(wall, 2), "successful_requests_per_s": round(rps, 3),
        "output_tokens_per_s": round(out_tps, 1),
        "ttft_p50_s": round(pct([r["ttft"] for r in ok], 0.5), 3), "ttft_p95_s": round(pct([r["ttft"] for r in ok], 0.95), 3),
        "e2e_p95_s": round(pct([r["e2e"] for r in ok], 0.95), 3),
        "price_per_hour_usd": price,
        "estimated_compute_cost_per_1000_successful_requests": round(price / (rps * 3600) * 1000, 5) if rps else None,
    }
 
 
def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--requests", type=int, default=64, help="requests per concurrency level")
    ap.add_argument("--concurrency", default="1,4,8,16", help="comma-separated levels to sweep")
    ap.add_argument("--price-per-hour", type=float, required=True, help="USD per hour for the machine")
    ap.add_argument("--label", required=True, help="e.g. cpu-laptop, gpu-l4; printed with every number")
    ap.add_argument("--out", required=True)
    a = ap.parse_args()
    one(0)                                                    # warm-up request, not counted
    report = [run(a.requests, int(c), a.price_per_hour, a.label) for c in a.concurrency.split(",")]
    Path(a.out).parent.mkdir(parents=True, exist_ok=True)
    Path(a.out).write_text(json.dumps(report, indent=2))
    print(f"{'conc':>5}{'req/s':>8}{'tok/s':>8}{'TTFT p95':>10}{'e2e p95':>9}{'errors':>7}{'USD/1k req':>12}")
    for r in report:
        print(f"{r['concurrency']:>5}{r['successful_requests_per_s']:>8.1f}{r['output_tokens_per_s']:>8.0f}"
              f"{r['ttft_p95_s']:>10.3f}{r['e2e_p95_s']:>9.3f}{r['errors']:>7}"
              f"{r['estimated_compute_cost_per_1000_successful_requests']:>12.5f}")
    print(f"label: {a.label} (compute-only estimate at {a.price_per_hour} USD/h)")
    return 0
 
 
if __name__ == "__main__":
    sys.exit(main())

Step 2: Run the sweep

Bash

python bench/bench.py --requests 48 --concurrency 1,4,8,16 \
  --price-per-hour <your machine USD/h> --label "cpu, <cpu model>, <cores>, <RAM>" --out runs/bench-cpu.json

Expected output (real run, but against the stand-in at an illustrative USD 0.0832 per hour: it shows the format and the arithmetic, NOT vLLM's speed):

TEXT

 conc   req/s   tok/s  TTFT p95  e2e p95 errors  USD/1k req
    1     7.8      78     0.065    0.139      0     0.00297
    4    26.1     261     0.109    0.197      0     0.00089
    8    41.9     419     0.123    0.196      0     0.00055
   16    36.7     367     0.426    0.493      0     0.00063
label: stand-in, NOT vLLM (compute-only estimate at 0.0832 USD/h)

Read it the way you will read your vLLM run: throughput rises with concurrency until the server saturates, then TTFT jumps while throughput stalls or drops (here between 8 and 16). The useful operating point is the highest concurrency whose TTFT p95 still meets your chat budget, not the peak throughput. Quote cost at that point. With vLLM on a laptop CPU, expect latencies in seconds and far lower throughput.

Acceptance check

runs/bench-cpu.json has 0 errors at your chosen operating point, with the hardware label and hourly price.

Milestone 5: Compare CPU and GPU Economics

This milestone repeats the sweep on a GPU (optional, paid) and compares at the same latency budget.

Step 1: Run the GPU path and fill the table

Bash

docker compose -f compose.yaml -f compose.gpu.yaml up -d vllm
python bench/bench.py --requests 48 --concurrency 1,4,8,16,32 --price-per-hour <GPU USD/h> \
  --label "gpu, <GPU model>" --out runs/bench-gpu.json

Option	Concurrency at TTFT p95 ≤ your budget	Estimated compute USD per 1,000 requests
CPU (your machine)	yours	yours
GPU (cloud, hourly)	yours	yours
Hosted API	n/a	price page × your mean tokens per request

Compare at the same latency budget, not at maximum throughput. Stop and delete the GPU instance and its disk the same day.

Acceptance check

The table is filled from your own runs, each row labelled with hardware, and the hosted-API row cites the price page and date.

Milestone 6: Gate Changes in CI with a Trusted Harness

This milestone runs the golden-set gate on every relevant pull request. The scoring code, the golden set and the gate come from the base commit, so a pull request cannot weaken how it is judged.

What changed from the MLflow module's gate

Trusted harness: run_golden.py, scoring.py, gate.py and golden/support.jsonl are taken from the base commit. The pull request supplies only the assistant code, prompts and knowledge under test (--app-dir).
Real baseline: the baseline is the base commit's app and its active prompt file, not a version name read from the base and then loaded from the pull request.
Immutable prompts: any edit to a published prompts/vN.yaml fails. A change means a new version file plus an edit to prompts/ACTIVE.
Invented claims: a passing answer must state the required facts and contain no claim from FORBIDDEN_CLAIMS. "Correct fact plus an invented loyalty bonus" now fails. refusal_accuracy is renamed out_of_scope_refusal_rate, which is what it measures.

The remaining trust gap, stated plainly: the candidate's assistant code runs inside the job, so it could in principle detect golden questions and return canned answers. The job has no secrets, and code review covers that risk; a prompt-only pipeline that never runs candidate code is stricter.

Step 1: The harness

eval/scoring.py:

Python

"""Deterministic scoring for policy answers. Lives with the trusted harness.
 
An answer PASSES an in-scope case only if it states every required fact, does not refuse,
and makes no claim from the forbidden list (policies acme-shop does not have). An
out-of-scope case passes only if the model refuses and also invents nothing.
"""
REFUSAL = ("don't know", "do not know", "contact support", "not sure", "cannot find")
 
# Claims that are NOT acme-shop policy. Extend this list whenever review finds a new invention.
FORBIDDEN_CLAIMS = ("loyalty point", "zero interest", "no-cost emi", "emi", "price match",
                    "ship to singapore", "international shipping", "no extra cost", "free returns on gift cards",
                    "lifetime warranty")
 
 
def score(answer: str, must_include: list[str], must_refuse: bool) -> dict:
    text = answer.lower()
    refused = any(p in text for p in REFUSAL)
    invented = [c for c in FORBIDDEN_CLAIMS if c in text]
    if must_refuse:
        passed = refused and not invented
    else:
        passed = not refused and not invented and all(f.lower() in text for f in must_include)
    return {"passed": passed, "refused": refused, "invented_claims": invented}

eval/run_golden.py:

Python

"""Score one prompt version of one assistant on the golden set. Part of the TRUSTED harness.
 
--app-dir chooses which assistant code, prompts and knowledge to run (base or candidate);
the golden set and the scoring rules always come from this harness's own directory.
Writes a JSON report and logs metrics plus trace ids to MLflow.
"""
import argparse
import json
import math
import os
import sys
from pathlib import Path
 
import mlflow
 
HARNESS = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(HARNESS / "eval"))
from scoring import score  # noqa: E402
 
 
def p95(xs):
    s = sorted(xs)
    return s[max(0, math.ceil(0.95 * len(s)) - 1)]
 
 
def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--prompt", required=True)
    ap.add_argument("--app-dir", default=".", help="where app/, prompts/ and knowledge/ live")
    ap.add_argument("--out", required=True)
    a = ap.parse_args()
    cases = [json.loads(l) for l in open(HARNESS / "golden/support.jsonl")]   # golden set from the harness
    out = Path(a.out).resolve()
    os.chdir(a.app_dir)                       # the assistant reads prompts/ and knowledge/ relative to here
    sys.path.insert(0, os.getcwd())
    from app.assistant import MODEL, answer   # the assistant under test
 
    mlflow.set_experiment("acme-support-eval")
    rows = []
    with mlflow.start_run(run_name=f"golden-{a.prompt}"):
        for c in cases:
            r = answer(c["question"], a.prompt)
            rows.append({**c, "answer": r["text"], **score(r["text"], c["must_include"], c["must_refuse"]),
                         "latency_s": r["latency_s"], "completion_tokens": r["completion_tokens"],
                         "trace_id": mlflow.get_last_active_trace_id()})
        oos = [r for r in rows if r["must_refuse"]]
        summary = {
            "prompt": a.prompt, "model": MODEL, "cases": len(rows),
            "pass_rate": sum(r["passed"] for r in rows) / len(rows),
            "out_of_scope_refusal_rate": sum(r["passed"] for r in oos) / len(oos),
            "answers_with_invented_claims": sum(bool(r["invented_claims"]) for r in rows),
            "p95_latency_s": p95([r["latency_s"] for r in rows]),
            "mean_completion_tokens": sum(r["completion_tokens"] for r in rows) / len(rows),
        }
        mlflow.log_params({"prompt_version": a.prompt, "model": MODEL})
        mlflow.log_metrics({k: float(v) for k, v in summary.items() if isinstance(v, (int, float)) and k != "cases"})
    out.parent.mkdir(exist_ok=True)
    out.write_text(json.dumps({"summary": summary, "cases": rows}, indent=2))
    for r in rows:
        if not r["passed"]:
            print(f"  FAIL {r['id']}: {r['answer'][:80]!r} invented={r['invented_claims']}")
    print(f"{a.prompt}: pass_rate={summary['pass_rate']:.3f} out_of_scope_refusal_rate="
          f"{summary['out_of_scope_refusal_rate']:.2f} invented={summary['answers_with_invented_claims']} "
          f"p95={summary['p95_latency_s']:.3f}s")
    return 0
 
 
if __name__ == "__main__":
    sys.exit(main())

eval/gate.py:

Python

"""Release gate. Rules written before any candidate ran:
 
* no answer may contain an invented claim, and every out-of-scope question must be refused
* pass_rate >= 0.85 and >= baseline - 0.05
* p95 latency <= 1.5 x baseline (catches a change that makes answers much longer)
Exit codes: 0 = pass, 1 = fail, 4 = bad input.
"""
import argparse
import json
import sys
from pathlib import Path
 
ap = argparse.ArgumentParser()
ap.add_argument("--candidate", required=True)
ap.add_argument("--baseline", required=True)
a = ap.parse_args()
try:
    c = json.loads(Path(a.candidate).read_text())["summary"]
    b = json.loads(Path(a.baseline).read_text())["summary"]
except (FileNotFoundError, KeyError) as err:
    print(f"ERROR: {err}"); sys.exit(4)
checks = [
    ("no invented claims", c["answers_with_invented_claims"] == 0, f"{c['answers_with_invented_claims']} answers"),
    ("refuses every out-of-scope question", c["out_of_scope_refusal_rate"] == 1.0,
     f"out_of_scope_refusal_rate {c['out_of_scope_refusal_rate']:.2f}"),
    ("pass rate floor", c["pass_rate"] >= 0.85, f"{c['pass_rate']:.3f} vs 0.85"),
    ("not worse than baseline", c["pass_rate"] >= b["pass_rate"] - 0.05,
     f"{c['pass_rate']:.3f} vs {b['pass_rate'] - 0.05:.3f} (baseline {b['prompt']})"),
    ("latency", c["p95_latency_s"] <= 1.5 * b["p95_latency_s"],
     f"p95 {c['p95_latency_s']:.3f}s vs {1.5 * b['p95_latency_s']:.3f}s"),
]
for name, ok, detail in checks:
    print(f"{'PASS' if ok else 'FAIL'}  {name}: {detail}")
passed = all(ok for _, ok, _ in checks)
print("GATE PASSED" if passed else "GATE FAILED")
sys.exit(0 if passed else 1)

Step 2: The workflow

.github/workflows/llm-eval.yml (not run for this guide: no GitHub access from the sandbox):

YAML

name: llm-eval
on:
  pull_request:
    paths: ["prompts/**", "app/**", "knowledge/**", "eval/**", "golden/**"]
 
permissions:
  contents: read                     # no secrets: the model is public and runs inside the job
 
jobs:
  golden-gate:
    runs-on: ubuntu-24.04
    timeout-minutes: 45
    env:
      LLM_BASE_URL: http://127.0.0.1:8000/v1
      MLFLOW_TRACKING_URI: sqlite:///mlflow.db
      MLFLOW_DISABLE_TELEMETRY: "true"
    steps:
      # Pinned by commit SHA: checkout v7.0.1, setup-python v7.1.0, upload-artifact v7.0.2
      - name: Check out the pull request (candidate)
        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
        with:
          path: candidate
          persist-credentials: false
      - name: Check out the base commit (trusted harness and baseline app)
        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
        with:
          ref: ${{ github.event.pull_request.base.sha }}
          path: base
          persist-credentials: false
      - uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0
        with:
          python-version: "3.12"
      - run: pip install -r base/requirements.txt          # the base's pinned libraries
      - name: Published prompt versions are immutable
        run: |
          for f in base/prompts/v*.yaml; do
            cmp -s "$f" "candidate/prompts/$(basename "$f")" || { echo "edited existing $(basename "$f"): add a new version instead"; exit 1; }
          done
      - name: Start vLLM on CPU (same image and flags as compose.yaml)
        run: |
          docker run -d --name vllm --shm-size 2g --memory 6g -p 8000:8000 vllm/vllm-openai-cpu:v0.33.0-x86_64 \
            --model=Qwen/Qwen2.5-0.5B-Instruct --served-model-name=acme-support \
            --max-model-len=4096 --kv-cache-memory-bytes=2147483648 --dtype=bfloat16
          for i in $(seq 1 90); do curl -sf localhost:8000/health && exit 0; sleep 10; done
          docker logs vllm | tail -50; exit 1
      - name: Score base (app + prompt) and candidate (app + prompt) with the BASE harness
        run: |
          BASE_PROMPT=$(cat base/prompts/ACTIVE)
          CAND_PROMPT=$(cat candidate/prompts/ACTIVE)
          [[ "$CAND_PROMPT" =~ ^v[0-9]+$ ]] || { echo "bad prompt version"; exit 1; }
          python base/eval/run_golden.py --prompt "$BASE_PROMPT" --app-dir base --out runs/baseline.json
          python base/eval/run_golden.py --prompt "$CAND_PROMPT" --app-dir candidate --out runs/candidate.json
      - run: python base/eval/gate.py --candidate runs/candidate.json --baseline runs/baseline.json
      - if: always()
        uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9
        with:
          name: golden-eval-${{ github.event.pull_request.number }}
          path: runs/

Step 3: Run the same comparison locally

Bash

python eval/run_golden.py --prompt v1 --out runs/eval-v1.json
python eval/run_golden.py --prompt v2 --out runs/eval-v2.json
python eval/gate.py --candidate runs/eval-v2.json --baseline runs/eval-v1.json; echo "exit=$?"

Expected output (real run, stand-in):

TEXT

  FAIL refund-timing: "I don't know based on our policies. Please contact support." invented=[]
  FAIL electronics-return: "I don't know based on our policies. Please contact support." invented=[]
v1: pass_rate=0.875 out_of_scope_refusal_rate=1.00 invented=0 p95=0.358s
  FAIL refund-timing: 'Yes! We are happy to offer that at no extra cost on every order.' invented=['no extra cost']
  FAIL electronics-return: 'Yes! We are happy to offer that at no extra cost on every order.' invented=['no extra cost']
  FAIL oos-loyalty: 'Yes! We are happy to offer that at no extra cost on every order.' invented=['no extra cost']
  FAIL oos-international: 'Yes! We are happy to offer that at no extra cost on every order.' invented=['no extra cost']
  FAIL oos-emi: 'Yes! We are happy to offer that at no extra cost on every order.' invented=['no extra cost']
  FAIL oos-price-match: 'Yes! We are happy to offer that at no extra cost on every order.' invented=['no extra cost']
v2: pass_rate=0.625 out_of_scope_refusal_rate=0.00 invented=6 p95=0.357s
FAIL  no invented claims: 6 answers
FAIL  refuses every out-of-scope question: out_of_scope_refusal_rate 0.00
FAIL  pass rate floor: 0.625 vs 0.85
FAIL  not worse than baseline: 0.625 vs 0.825 (baseline v1)
PASS  latency: p95 0.357s vs 0.537s
GATE FAILED
exit=1

The two v1 failures are retrieval misses: "When will my refund reach my card?" shares only one word with the right policy line, below the two-word relevance threshold. That is the threshold's trade-off, visible in the golden set. Latency passed for v2, so a latency-only check would have shipped it.

Step 4: Tests

tests/test_gate.py:

Python

"""Scoring, gate rules, retrieval and redaction, no model needed."""
import json
import os
import subprocess
import sys
 
os.environ.setdefault("MLFLOW_TRACKING_URI", "sqlite:///test-mlflow.db")
sys.path.insert(0, "eval")
from app.assistant import retrieve  # noqa: E402
from app.redact import redact  # noqa: E402
from scoring import score  # noqa: E402
 
 
def gate(tmp_path, cand: dict, base: dict) -> int:
    for name, s in (("c", cand), ("b", base)):
        (tmp_path / f"{name}.json").write_text(json.dumps({"summary": {"prompt": name, **s}}))
    return subprocess.run([sys.executable, "eval/gate.py", "--candidate", str(tmp_path / "c.json"),
                           "--baseline", str(tmp_path / "b.json")], capture_output=True).returncode
 
 
BASE = {"pass_rate": 0.875, "out_of_scope_refusal_rate": 1.0, "answers_with_invented_claims": 0, "p95_latency_s": 1.0}
 
 
def test_equal_candidate_passes(tmp_path):
    assert gate(tmp_path, BASE, BASE) == 0
 
 
def test_one_invented_claim_fails_even_with_a_high_pass_rate(tmp_path):
    assert gate(tmp_path, {**BASE, "pass_rate": 0.95, "answers_with_invented_claims": 1}, BASE) == 1
 
 
def test_much_slower_candidate_fails(tmp_path):
    assert gate(tmp_path, {**BASE, "p95_latency_s": 1.6}, BASE) == 1
 
 
def test_correct_fact_plus_invented_policy_fails():
    good = score("Refunds are paid within 5 to 7 business days.", ["5 to 7 business days"], False)
    padded = score("Refunds are paid within 5 to 7 business days, and you also earn loyalty points!",
                   ["5 to 7 business days"], False)
    assert good["passed"] and not padded["passed"] and padded["invented_claims"] == ["loyalty point"]
 
 
def test_refusal_that_also_invents_fails():
    assert not score("I don't know, but we offer zero interest EMI.", [], True)["passed"]
 
 
def test_off_topic_question_retrieves_nothing():
    assert retrieve("Do you ship to Singapore?") == []
    assert any("30 days" in line for line in retrieve("How many days do I have to return an item for a refund?"))
 
 
def test_redaction_removes_personal_data():
    out = redact("I'm priya.k@example.com, order ACME-1234567, phone +91 98765 43210, card 4111 1111 1111 1111")
    assert out == "I'm [EMAIL], order [ORDER], phone [PHONE], card [CARD]"

Expected output (real run): 7 passed.

Acceptance check

A pull request that sets prompts/ACTIVE to v2 fails golden-gate; one that edits prompts/v1.yaml fails the immutability step.

Milestone 7: Monitor Live Answer Quality by Prompt Version

This milestone turns production traces into a quality signal with no labels: refusal rate and invented-claim rate per prompt version and model, plus a review queue for humans.

Step 1: Sample the traces

quality/sample_review.py:

Python

"""Live answer quality without labels: read recent traces, group by prompt version and model,
flag refusals and invented claims, and write a review queue for humans.
 
Uses the same FORBIDDEN_CLAIMS list as the CI gate, so a claim found in review is added once
and then guarded everywhere.
"""
import csv
import random
import sys
from collections import defaultdict
 
import mlflow
 
sys.path.insert(0, "eval")
from scoring import FORBIDDEN_CLAIMS, REFUSAL  # noqa: E402
 
exp = mlflow.get_experiment_by_name("acme-support-live")
traces = mlflow.search_traces(locations=[exp.experiment_id], max_results=1000, return_type="list")
groups, queue = defaultdict(lambda: {"n": 0, "refused": 0, "invented": 0}), []
for t in traces:
    tags = t.info.tags
    key = (tags.get("prompt_version", "?"), tags.get("model", "?"))
    root = next(s for s in t.data.spans if s.parent_id is None)
    text = str((root.outputs or {}).get("text", "")).lower()
    question = str((root.inputs or {}).get("question", ""))
    refused, invented = any(p in text for p in REFUSAL), [c for c in FORBIDDEN_CLAIMS if c in text]
    g = groups[key]
    g["n"] += 1; g["refused"] += refused; g["invented"] += bool(invented)
    if invented or random.random() < 0.05:                     # every invention + a 5% random sample
        queue.append({"trace_id": t.info.trace_id, "prompt_version": key[0], "question": question,
                      "answer": text[:200], "invented": ";".join(invented)})
print(f"{'prompt':<8}{'model':<14}{'requests':>9}{'refusal rate':>14}{'invented rate':>15}")
for (pv, model), g in sorted(groups.items()):
    print(f"{pv:<8}{model:<14}{g['n']:>9}{g['refused'] / g['n']:>14.2f}{g['invented'] / g['n']:>15.2f}")
with open("review_queue.csv", "w", newline="") as f:
    w = csv.DictWriter(f, fieldnames=["trace_id", "prompt_version", "question", "answer", "invented"])
    w.writeheader(); w.writerows(queue)
print(f"review_queue.csv: {len(queue)} traces for human review")

Bash

python quality/sample_review.py

Expected output (real run, stand-in, after Milestone 2's 200 requests):

TEXT

prompt  model          requests  refusal rate  invented rate
v1      acme-support        179          0.30           0.00
v2      acme-support         21          0.00           0.33
review_queue.csv: 15 traces for human review

The canary prompt v2 invents a policy in a third of its answers while v1 invents none. That is the production view of what the CI gate caught. If a change ever reaches production anyway, this signal is how you see it within hours, before chargebacks or complaints. It connects serving to quality through the prompt_version and model tags on every trace.

Acceptance check

The table separates prompt versions, and review_queue.csv holds every answer with an invented claim plus a random sample.

Common Mistake
Reading a falling refusal rate as good news. After a prompt change it often means the model stopped admitting it does not know. Read it next to the invented-claim rate.

Break It on Purpose

Step 1: Edit a published prompt instead of adding a version

Bash

git worktree add /tmp/base HEAD             # the base commit, side by side, as CI checks it out
sed -i 's/Keep answers under 60 words./Keep answers short and friendly./' prompts/v1.yaml
for f in /tmp/base/prompts/v*.yaml; do
  cmp -s "$f" "prompts/$(basename "$f")" || { echo "edited existing $(basename "$f"): add a new version instead"; break; }
done

Expected output (real run, with the base and candidate checkouts side by side as in CI):

TEXT

edited existing v1.yaml: add a new version instead

Undo the edit with git checkout prompts/v1.yaml.

Step 2: Remove the retrieval threshold

Change min_overlap: int = 2 to 1 in retrieve and rerun v1. Observed while building this project, before the threshold existed (real run, stand-in):

TEXT

  FAIL oos-loyalty: 'Standard delivery takes 3 to 5 business days and is free on orders above 999 rupees. Cash '
  FAIL oos-emi: 'Refunds are accepted within 30 days of delivery for unused items with the original receipt'
v1: pass_rate=0.875 refusal_accuracy=0.50 p95=0.403s tokens=33.5

Off-topic questions pulled in unrelated policies, and with context in hand the "model" answered instead of refusing. (That run used the old metric name.) Put the threshold back.

Step 3: Starve the KV cache (needs vLLM)

Set --kv-cache-memory-bytes=268435456 (256 MiB), restart, and run the sweep up to 16.

Expected behaviour (not run): vllm:kv_cache_usage_perc near 1.0, vllm:num_requests_waiting above zero, TTFT rising sharply at lower concurrency than before, possibly preemptions. RequestsQueueing and KvCacheNearlyFull turn firing in Prometheus. Restore 2 GiB.

Validation & Testing

Step 1: Final checks

Bash

pytest -q                                   # 7 passed
./monitoring/check_metrics.sh               # 5 found (needs vLLM)
python quality/sample_review.py             # per-version quality table
cat runs/bench-*.json

Definition of done

vLLM runs on CPU under a hard memory limit, and the README records machine spec and docker stats memory.
Prometheus loads the five rules, and check_metrics.sh confirms every metric on your server; alert delivery is documented as out of scope.
The concurrency sweep names an operating point and an estimated compute cost per 1,000 successful requests, labelled as compute-only.
Stored traces contain no raw personal data, and old traces are deleted on a schedule.
The CI gate uses the base commit's harness and baseline, published prompts are immutable, and invented claims fail.
Live quality is reported per prompt version and model, with a review queue.
Every number in the README says CPU or GPU, and none comes from the stand-in.

Step 2: Tear down

Bash

docker compose down
kill %1 2>/dev/null                          # the stand-in, if running
rm -rf runs mlflow.db review_queue.csv
docker image rm vllm/vllm-openai-cpu:v0.33.0-x86_64

Present This Project in an Interview

The 60-second version

"I served Qwen2.5-0.5B with vLLM's OpenAI-compatible server on CPU, with a hard memory limit sized from weights plus KV cache plus runtime, and a GPU override. Prometheus watches TTFT, the waiting queue, KV cache usage and preemptions, and I check that those metric names exist on the running server, not just that the rules parse. A streaming benchmark sweeps concurrency, and I quote cost at the highest concurrency that still meets the TTFT budget, as a compute-only estimate. For quality, the assistant redacts personal data before tracing, CI judges changes with the base commit's harness so a pull request cannot weaken its own gate, and a live sampler reports invented-claim rate per prompt version. A canary prompt that invented policies in a third of its answers showed up there immediately."

Questions to expect

"Why is the cost only an estimate?" It is hourly price over measured throughput at full utilisation. Idle time, storage, network and people are not in it, and at low traffic the real cost per request is much higher.
"What actually limits concurrency?" KV cache. Each running request holds cache for its tokens, so --max-model-len and the cache size set how many fit before requests queue.
"How do you stop a pull request gaming the eval?" The harness, golden set and gate come from the base commit and published prompts are immutable. The candidate's code still runs, so review covers the rest.
"How do you monitor quality without labels?" Deterministic signals on every trace (refusals, forbidden claims), grouped by prompt version and model, plus a sampled human review queue.

What to show on screen

The concurrency sweep with your hardware label and chosen operating point.
check_metrics.sh output and the Prometheus targets page.
The per-prompt-version quality table.

Found a mistake, an outdated step or a broken link?

Serve an LLM with vLLM and Monitor Cost and Quality

Blueprint Walkthrough

Architecture Overview & Problem Statement

The problem

What you build

How a change is judged in CI

Repository layout

Prerequisites (by module title)

Versions used

Milestone 1: Serve Qwen2.5-0.5B with vLLM on CPU

Where the memory goes

Step 1: Write the Compose files

Step 2: Start and check

Acceptance check

Milestone 2: Connect the Assistant, Redacted and Traced

Step 1: Reuse the assistant and add redaction

Step 2: The offline stand-in (for when you have no model running)

Step 3: Send traffic and check the stored traces

Acceptance check

Milestone 3: Monitor vLLM with Prometheus

Step 1: Configure Prometheus

Step 2: Prove the metrics exist on YOUR server

What each metric tells you

Acceptance check

Milestone 4: Sweep Concurrency and Estimate Compute Cost

What the cost number is, and is not

Step 1: Write the benchmark

Step 2: Run the sweep

Acceptance check

Milestone 5: Compare CPU and GPU Economics

Step 1: Run the GPU path and fill the table

Acceptance check

Milestone 6: Gate Changes in CI with a Trusted Harness

What changed from the MLflow module's gate

Step 1: The harness

Step 2: The workflow

Step 3: Run the same comparison locally

Step 4: Tests

Acceptance check

Milestone 7: Monitor Live Answer Quality by Prompt Version

Step 1: Sample the traces

Acceptance check

Break It on Purpose

Step 1: Edit a published prompt instead of adding a version

Step 2: Remove the retrieval threshold

Step 3: Starve the KV cache (needs vLLM)

Validation & Testing

Step 1: Final checks

Definition of done

Step 2: Tear down

Present This Project in an Interview

The 60-second version

Questions to expect

What to show on screen