A RAG chatbot works perfectly on your laptop. You ask it a question, it retrieves the right chunks, the model responds in two seconds, everyone in the demo is impressed. Three weeks later it is live for 400 Zerodha support agents, and by 11 AM the response times have crawled past 30 seconds, nobody can tell why the bill jumped to 40,000 rupees overnight, and a single slow request is holding a database connection open long enough to start queuing every request behind it. None of the model's reasoning got worse. What changed is that a prototype only has to work once, in front of you, on a good day. A production system has to work every time, for every user, including the ones who send malformed input, the ones on a bad network connection, and the ones hitting it at the exact same second as five hundred other people. **LLMOps** is the discipline of taking an LLM application from a working prototype to a system that survives that. It covers how you serve the model over an API, how you contain and deploy that service reliably, how you know what it is doing at any given moment, and how you keep its cost from growing faster than your business does. > 📌 **Remember:** A prototype proves the idea works. Production engineering is the separate, substantial body of work that makes it keep working under real traffic, real failures, and real cost pressure. ### Where this module fits in the stack you have already built Earlier modules covered the pieces this system is made of. LLM Fundamentals covered calling a model. Embeddings and RAG covered retrieving the right context. Agents and Reliability covered making a multi-step call sequence survive failure. Evaluation covered proving the system's outputs are actually good. This module is what wraps all of that in something you can actually hand a URL to and let real users hit. If the RAG chatbot from the RAG module is the brain, this module is the nervous system, the skeleton, and the immune system around it, the parts that keep it alive under load rather than the parts that make it smart. ### The shape of a production AI system Before going section by section, it helps to see the whole shape at once. Every piece covered in this module fits into this same picture. Client (app / user) | v API Layer (FastAPI + auth + rate limiting) | v Cache check (prompt cache / semantic cache) hit --> return cached response miss | v Model call (managed API or self-hosted, containerized, on Kubernetes) | v Observability (traces, spans, cost, eval scores attached to the trace) | v Response streamed back to client This diagram is the map for this entire module. Each section below builds one box in it, in roughly the order a request actually flows through them, ending with the observability layer that watches every other box at once. ---
Whatever else your AI application does, at some point a user's request needs to reach your code over HTTP, and your response needs to go back out. **FastAPI** is the standard choice for this in Python, and the reason it fits LLM applications specifically, not just APIs in general, is worth understanding before you write a line of it. ### Why FastAPI specifically for AI services FastAPI is built around **Pydantic**, a library for defining the exact shape of data you expect, and validating incoming requests against that shape automatically before your code ever runs. For an AI service this matters more than for a typical CRUD API, because LLM requests carry a lot of easy-to-miss detail, a `temperature` that should be between 0 and 2, a `max_tokens` that should be a positive integer, a `messages` array that should never be empty. Catching a malformed request before it reaches the model saves you a wasted, billed API call to the LLM provider for a request that was never going to work. FastAPI also has first-class async support, which matters because a call to an LLM provider spends most of its time waiting on the network, not doing CPU work. An async endpoint can hold hundreds of in-flight LLM calls at once without needing hundreds of OS threads. ```python from fastapi import FastAPI, HTTPException from pydantic import BaseModel, Field app = FastAPI() class ChatRequest(BaseModel): """ Defines exactly what a valid chat request looks like. FastAPI validates every incoming request against this before your endpoint function ever runs, so a malformed request never reaches the model. """ message: str = Field(..., min_length=1, max_length=4000) session_id: str temperature: float = Field(default=0.7, ge=0.0, le=2.0) max_tokens: int = Field(default=512, gt=0, le=4096) class ChatResponse(BaseModel): reply: str session_id: str tokens_used: int @app.post("/chat", response_model=ChatResponse) async def chat(request: ChatRequest) -> ChatResponse: # By the time this line runs, request.temperature is guaranteed # to be a float between 0.0 and 2.0, request.message is guaranteed # non-empty. FastAPI already rejected anything that didn't match. reply, tokens = await call_llm(request.message, request.temperature) return ChatResponse(reply=reply, session_id=request.session_id, tokens_used=tokens) ``` > **Note:** `Field(..., min_length=1, max_length=4000)` means this field is required (the `...`) and must be between 1 and 4000 characters. Pydantic rejects the request with a clear error before `chat()` runs if that is violated, the same way `ge=0.0, le=2.0` on `temperature` rejects a value outside that range. ### Async endpoints and why blocking calls silently ruin them Writing `async def` on an endpoint does not automatically make everything inside it non-blocking. If you call a synchronous, blocking function inside an `async def` endpoint, you block the entire event loop, every other request being handled by that worker stalls until the blocking call finishes. ```python import httpx # WRONG - requests is a synchronous library, this blocks the whole event loop @app.post("/chat-blocking") async def chat_blocking(request: ChatRequest): import requests response = requests.post("https://api.llm-provider.com/v1/chat", json={...}) # every other in-flight request on this worker is frozen until this returns return response.json() # CORRECT - httpx is async-native, this actually yields control back @app.post("/chat-async") async def chat_async(request: ChatRequest): async with httpx.AsyncClient() as client: response = await client.post("https://api.llm-provider.com/v1/chat", json={...}) # other requests on this worker are served while this awaits return response.json() ``` > 🔴 **Common Mistake:** Marking an endpoint `async def` and assuming that alone makes it non-blocking. If any line inside calls a synchronous library, database driver, or file operation without `await`, that call blocks the entire event loop for every concurrent request on that worker, not just this one. The fix: use async-native libraries (`httpx` instead of `requests`, an async database driver) for anything inside an `async def` endpoint, or explicitly run blocking code in a thread pool with `run_in_threadpool`. ---
A user asking a chatbot a question does not want to stare at a blank screen for eight seconds before the full answer appears. **Streaming** sends the model's output token by token as it is generated, so the user sees the response forming in real time, the same experience you get from ChatGPT's interface. ### Why streaming matters more for LLMs than for typical APIs Most APIs return in milliseconds, streaming would add complexity for no benefit. LLM responses are different, they can take anywhere from one to thirty seconds depending on length and load, and that gap is long enough that a non-streaming response feels broken even when it is working correctly. Streaming does not make the total generation time faster, it makes the wait feel shorter because the user sees progress immediately. ```python from fastapi.responses import StreamingResponse import json @app.post("/chat-stream") async def chat_stream(request: ChatRequest): async def token_generator(): async for token in call_llm_streaming(request.message, request.temperature): # Server-Sent Events format - each chunk is prefixed with "data: " yield f"data: {json.dumps({'token': token})}\n\n" yield "data: [DONE]\n\n" return StreamingResponse(token_generator(), media_type="text/event-stream") ``` > **Note:** `text/event-stream` is the MIME type for Server-Sent Events, a simple standard for a server pushing a sequence of small text updates to a client over one open connection. The client library on the frontend reads each `data: ...` chunk as it arrives and appends it to the visible response, which is what produces the token-by-token typing effect. ### A hung request is as bad as a failed one A request that never returns and is never cancelled consumes a connection, a worker thread or task slot, and often a paid model call, indefinitely, while telling the user and your monitoring nothing useful. This applies to both streaming and non-streaming calls, and it applies just as much to an agent run, covered in the Agents and Reliability module, as to a single model call. ```python import asyncio @app.post("/chat-with-timeout") async def chat_with_timeout(request: ChatRequest): try: # Wait at most 30 seconds for the full response. If the model # provider hangs or the network stalls, this raises instead of # holding the connection open forever. reply = await asyncio.wait_for( call_llm(request.message, request.temperature), timeout=30.0 ) return {"reply": reply} except asyncio.TimeoutError: # The client gets a clear, immediate answer instead of a silent hang raise HTTPException(status_code=504, detail="Model response timed out") ``` For a streaming response, the same idea applies per-chunk rather than to the whole call, if no new token arrives within a set window, the connection should be closed rather than left open indefinitely. And exactly as covered in the Agents and Reliability module, a timeout tells you a response did not arrive in time, it does not tell you whether the underlying operation actually completed or failed, that same caution about retries applies here if a request times out and your client considers resubmitting it. > 📌 **Engineering Decision:** Set an explicit timeout on every external call your service makes, the LLM provider call, any retrieval call, any tool call, never rely on the client's own timeout as your only protection. A hung upstream call should fail loudly and quickly on your side, not tie up your service's resources indefinitely. ---
An LLM-backed API is a more attractive and more expensive target than a typical CRUD API. Every request you serve can cost real money in model tokens, and if your API can trigger an agent that takes real actions, covered in the Agents and Reliability module, an unauthenticated or unthrottled endpoint is not just a cost risk, it is an action risk. ### Authentication: proving who is calling **Authentication** answers "who is making this request." The standard pattern for a service-to-service or app-to-backend API is an API key passed in a header, validated against a store of issued keys before the request is allowed to proceed. ```python from fastapi import Depends, Header, HTTPException async def verify_api_key(x_api_key: str = Header(...)) -> str: """ Validates the API key from the request header against issued keys. Returns the tenant_id associated with a valid key, so the endpoint can use it for authorization and rate limiting below. """ tenant_id = await api_key_store.lookup(x_api_key) if tenant_id is None: raise HTTPException(status_code=401, detail="Invalid API key") return tenant_id @app.post("/chat") async def chat(request: ChatRequest, tenant_id: str = Depends(verify_api_key)): # tenant_id is guaranteed valid by the time this line runs ... ``` > **Note:** `Depends(verify_api_key)` is FastAPI's dependency injection pattern. FastAPI runs `verify_api_key` before the endpoint body, and if it raises an exception, the endpoint body never executes at all. This is the same validate-before-execute principle covered for tool calls in the Agents and Reliability module, applied here at the API boundary instead of the tool boundary. ### Authorization and tenant isolation **Authorization** answers a separate question: "is this specific caller allowed to do this specific thing." For a multi-tenant AI application, for example a SaaS product serving many Swiggy-style restaurant partners from one backend, authorization also has to enforce **tenant isolation**, one tenant's request must never be able to read another tenant's data, conversation history, or documents, even by accident. ```python async def get_conversation(session_id: str, tenant_id: str = Depends(verify_api_key)): conversation = await db.get_conversation(session_id) if conversation is None: raise HTTPException(status_code=404, detail="Conversation not found") # Critical check: the conversation must belong to the calling tenant. # Without this, tenant A could read tenant B's data just by guessing # or enumerating session_id values. if conversation.tenant_id != tenant_id: raise HTTPException(status_code=403, detail="Not authorized for this conversation") return conversation ``` > 🔴 **Common Mistake:** Filtering by `tenant_id` in application code after fetching data, rather than as part of the database query itself. If a query fetches a record by `session_id` alone and only checks `tenant_id` afterward, a bug anywhere else in the code path that skips the check leaks cross-tenant data. The safer pattern is to include `tenant_id` directly in the database query's `WHERE` clause, so a record belonging to another tenant is never even fetched, not just filtered out afterward. ### Rate limiting **Rate limiting** caps how many requests, or how much cost, a given caller can generate in a time window. Without it, one runaway client, whether malicious or just buggy, a frontend stuck in a retry loop, can consume your entire model provider quota or run up an unbounded bill. ```python from fastapi import Request import time request_counts: dict[str, list[float]] = {} async def rate_limit(tenant_id: str = Depends(verify_api_key), max_per_minute: int = 60): """ Simple sliding-window rate limiter. Production systems typically use Redis for this instead of an in-process dict, so limits are enforced correctly across multiple API server instances, not just one. """ now = time.time() window_start = now - 60 recent = [t for t in request_counts.get(tenant_id, []) if t > window_start] if len(recent) >= max_per_minute: raise HTTPException(status_code=429, detail="Rate limit exceeded") recent.append(now) request_counts[tenant_id] = recent ``` > **Note:** This in-process dict works for a single server instance but breaks the moment you run more than one, each instance has its own separate count, so the real limit becomes `max_per_minute` multiplied by the number of instances. Production rate limiting is almost always backed by Redis or a similar shared store precisely so every instance enforces the same limit against the same counter. | Concern | Question It Answers | Failure Without It | |:---|:---|:---| | Authentication | Who is calling? | Anyone can call the API for free | | Authorization | Are they allowed to do this specific thing? | A caller can act outside their own scope | | Tenant isolation | Can they see another tenant's data? | Cross-customer data leak | | Rate limiting | How much can they call in a window? | Runaway cost or provider quota exhaustion | ---
Your FastAPI service now needs to run somewhere other than your own laptop, reliably, the same way every time, on a teammate's machine, a staging server, and production. **Docker** is how you package the service, its Python dependencies, and everything it needs to run into one portable unit that behaves identically everywhere. > This section covers Docker at the depth an AI service actually needs. For the complete general treatment of Docker, containers versus VMs, networking, and Compose, see the dedicated Docker module. ### Why AI services specifically need careful containerization A typical web service has a handful of small dependencies. An AI service commonly depends on heavy libraries, `torch`, `transformers`, `sentence-transformers`, an embedding model's weights, sometimes gigabytes of them. Get the Dockerfile wrong and every code change forces a multi-gigabyte dependency reinstall, turning a ten-second deploy into a fifteen-minute one. ```dockerfile # BAD - copies everything before installing dependencies FROM python:3.11-slim WORKDIR /app COPY . /app RUN pip install -r requirements.txt # Any change to any file, even a one-line code fix, invalidates the # cache for this RUN line, forcing torch and transformers to reinstall # GOOD - dependencies installed before the frequently-changing source code FROM python:3.11-slim WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt # This layer is cached and reused unless requirements.txt itself changes COPY . . CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"] ``` > **Note:** Docker caches each instruction in a Dockerfile as a layer. If a layer's inputs have not changed, Docker reuses the cached result instead of re-running it. Putting `COPY requirements.txt .` and the `pip install` before `COPY . .` means editing your application code never invalidates the expensive dependency-install layer, only editing `requirements.txt` does. ### Multi-stage builds for smaller AI service images An AI service's final image often only needs the installed Python packages and your application code at runtime, not the build tools, compilers, or intermediate files used to get there. A multi-stage build keeps those out of the final image entirely. ```dockerfile # Stage 1: install dependencies, including anything needing a compiler FROM python:3.11-slim AS builder WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir --user -r requirements.txt # Stage 2: copy only the installed packages and app code, nothing else FROM python:3.11-slim AS final WORKDIR /app COPY --from=builder /root/.local /root/.local COPY . . ENV PATH=/root/.local/bin:$PATH CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"] ``` The compiler toolchain and any build-only files used in Stage 1 never appear in the final image, only the installed packages copied across in `COPY --from=builder`. This is the same pattern used for a compiled Go binary in the general Docker module, applied here to a Python dependency install instead. ### A realistic AI service Dockerfile ```dockerfile FROM python:3.11-slim WORKDIR /app # System dependencies some AI libraries need to build against RUN apt-get update && apt-get install -y --no-install-recommends \ build-essential \ && rm -rf /var/lib/apt/lists/* COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY . . # Never run as root in production RUN useradd -m appuser USER appuser EXPOSE 8000 HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \ CMD curl -f http://localhost:8000/health || exit 1 CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"] ``` > 🔴 **Common Mistake:** Baking API keys or model provider credentials directly into the Dockerfile or the image with `ENV OPENAI_API_KEY=sk-...`. Anyone who can pull or inspect the image can read them out. The fix: inject secrets at container runtime through environment variables passed by the orchestrator, covered in the Kubernetes section below, or a secrets manager, never bake them into the image itself. ---
A single container running on a single machine works until traffic grows past what one machine can serve, or until that one machine has a problem and takes your entire service down with it. **Kubernetes** runs many copies of your container across many machines, restarts them automatically when they fail, and scales the number of copies up or down as demand changes. > This section covers the Kubernetes concepts an AI engineer actually touches day to day. For the complete treatment of Kubernetes architecture, StatefulSets, Ingress, RBAC, and Helm, see the dedicated Kubernetes module. ### Why an AI service specifically benefits from Kubernetes LLM traffic is often bursty rather than steady, quiet for hours, then a spike when a marketing email goes out or a feature launches. A fixed number of servers either sits idle most of the time or falls over during the spike. Kubernetes lets you declare "run between 2 and 10 copies of this service, add more when CPU or request volume rises" and it handles the mechanics, instead of you manually starting and stopping servers. The three Kubernetes building blocks that matter most for a service like this are a **Pod**, the smallest running unit, wrapping your container, a **Deployment**, which keeps a specified number of Pods running and replaces any that crash, and a **Service**, which gives that shifting set of Pods one stable network address other things can reliably call. Deployment (I want 3 replicas of my-service, always) | +-----+-----+-----+ | | | | Pod 1 Pod 2 Pod 3 <- your containerized FastAPI service | | | +-----+-----+ | Service <- one stable address, load balances across the 3 Pods ### A minimal Deployment for the FastAPI service ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: rag-chatbot-api spec: replicas: 3 selector: matchLabels: app: rag-chatbot-api template: metadata: labels: app: rag-chatbot-api spec: containers: - name: rag-chatbot-api image: myregistry/rag-chatbot-api:v1.2.0 ports: - containerPort: 8000 env: - name: OPENAI_API_KEY valueFrom: secretKeyRef: name: llm-provider-secret key: api-key resources: requests: memory: "256Mi" cpu: "250m" limits: memory: "512Mi" cpu: "500m" readinessProbe: httpGet: path: /health port: 8000 initialDelaySeconds: 5 periodSeconds: 5 livenessProbe: httpGet: path: /health port: 8000 initialDelaySeconds: 15 periodSeconds: 10 ``` > **Note:** `secretKeyRef` injects the API key from a Kubernetes Secret as an environment variable at runtime, this is the correct way to get the credential into the container without ever baking it into the image, following directly from the Docker security warning above. `readinessProbe` and `livenessProbe` follow the same pattern as any other service, if the health endpoint fails, the readiness probe pulls this Pod out of the Service's rotation without killing it, while the liveness probe restarts a Pod that has genuinely frozen. ```yaml apiVersion: v1 kind: Service metadata: name: rag-chatbot-api spec: selector: app: rag-chatbot-api ports: - port: 80 targetPort: 8000 type: ClusterIP ``` ### Resource requests for AI workloads specifically The `resources.requests` and `resources.limits` fields matter more for AI services than for a typical lightweight API, because loading an embedding model or holding a large in-memory cache can use meaningfully more memory than a simple CRUD service, and an underestimated memory limit gets your container killed with an `OOMKilled` error the moment it tries to load a model into memory. Size these based on what your service actually uses under real load, checked with `kubectl top pods` after deploying, not guessed. If your workload needs a GPU, for example a self-hosted model rather than a call to a managed provider, that is requested explicitly: ```yaml resources: limits: nvidia.com/gpu: 1 ``` Kubernetes then only schedules this Pod on a node that actually has a GPU available, the same node-matching mechanism covered in the Kubernetes module's section on taints, tolerations, and node affinity, applied here specifically to GPU nodes. ### Autoscaling under bursty AI traffic A **Horizontal Pod Autoscaler** watches a metric, commonly CPU, and adjusts the replica count within a defined range automatically. ```yaml apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: rag-chatbot-api-hpa spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: rag-chatbot-api minReplicas: 2 maxReplicas: 15 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 70 ``` For an AI service specifically, keep in mind that CPU usage often does not reflect the real bottleneck, most of a request's time is spent waiting on the model provider's response over the network, not doing CPU work locally. A custom metric based on request queue depth or in-flight request count is often a more accurate scaling signal for LLM-backed services than CPU alone, this is a known limitation worth flagging rather than a solved problem this module prescribes an exact answer for. > 📌 **Engineering Decision:** Start with CPU-based autoscaling because it is built in and simple, but if scaling reacts too slowly or too late during real traffic spikes, that is the signal to move to a custom metric tied to actual request concurrency or queue depth, not a reason to assume autoscaling itself is broken. ---
A RAG chatbot works perfectly on your laptop. You ask it a question, it retrieves the right chunks, the model responds i...
Whatever else your AI application does, at some point a user's request needs to reach your code over HTTP, and your resp...
A user asking a chatbot a question does not want to stare at a blank screen for eight seconds before the full answer app...
An LLM-backed API is a more attractive and more expensive target than a typical CRUD API. Every request you serve can co...
Your FastAPI service now needs to run somewhere other than your own laptop, reliably, the same way every time, on a team...
A single container running on a single machine works until traffic grows past what one machine can serve, or until that ...
Every AI service needs to decide how it actually calls a model, through a managed provider's API, or by running the mode...
Once your service is live, the question stops being "does it work" and becomes "what is it actually doing, for whom, and...
An LLM-backed service's per-request cost is not fixed the way a typical API's is, every token generated has a real, mete...
> Note: These commands assume Docker and a local Kubernetes cluster (Minikube or Kind) are already installed, covered in...
Request flow decision table Stage What Runs Here Covered In API layer Auth, authorization, rate limiting Securing the AP...
[image-1] The production AI system request flow: [Style: Clean flat diagram on white background] -> [Title: "Request Flo...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.