Learn to deploy ML models on Kubernetes with KServe: InferenceService, custom predictors, autoscaling, canary rollouts, rollback and endpoint security.
At acme-shop the fraud team starts with one model and one hand-written Deployment.
Before installing anything, it helps to know what the pieces are and which mode you are choosing.
KServe has several dependencies, so installation is where most first attempts stall.
A deployed model needs to come from somewhere.
A custom predictor is an ordinary Python server that follows KServe's prediction protocol.
Autoscaling is where KServe saves the most pain, and where wrong numbers cost the most.
At acme-shop the fraud team starts with one model and one hand-written Deployment.
Before installing anything, it helps to know what the pieces are and which mode you are choosing.
KServe has several dependencies, so installation is where most first attempts stall.
A deployed model needs to come from somewhere.
A custom predictor is an ordinary Python server that follows KServe's prediction protocol.
Autoscaling is where KServe saves the most pain, and where wrong numbers cost the most.
A canary rollout lets a small share of real requests reach the new model while most still go to the proven one.
A prediction endpoint is an API that returns decisions about customers. Treat it like one.
You cannot run what you cannot see. KServe gives you three windows: metrics, logs and, optionally, a stream of requests...
You will deploy acme-fraud v1 on KServe, validate the server locally, send a slice of traffic to v2, roll back, break...
Quick reference Common mistakes The first mistake is choosing raw (Standard) mode and then expecting a percentage...
Roles that use the skills in this module.
DevOps Engineer
Site Reliability Engineer
Platform Engineer
Not a software engineer sheet. Every problem comes from real DevOps, SRE, Platform and Cloud interviews, from your first script to a system you build yourself.
Open the Coding SheetUse KServe when you run several models and want one repeatable way to deploy, autoscale, split traffic and observe them. For a single model with steady traffic, a Deployment, a Service and a HorizontalPodAutoscaler is simpler and has fewer moving parts. Add KServe when the copy-pasted YAML starts to hurt.
In KServe v0.16, canaryTrafficPercent is supported only in serverless mode because it relies on Knative revisions. Standard mode requires you to manage traffic splitting separately. Newer KServe releases introduce additional Standard-mode canary capabilities through a different API, so always check the rollout documentation for your pinned version.
A pickled model remembers the classes it was built from. If it contains a custom class such as a wrapper from your own package, the runtime cannot rebuild it unless that package is installed. Either keep models to standard library classes or build a custom predictor image that includes your code.
With the canary strategy, KServe tracks the last revision that was rolled out successfully and keeps sending traffic there. A revision that never becomes ready should not receive traffic. Do not rely on that alone: check the actual traffic split and send test requests, and alert on readiness.