Learn to trace LLM and RAG calls, version prompts with aliases, gate prompt changes in CI and roll back safely using MLflow 3.
On a Friday afternoon at acme-shop, someone edits the support chatbot's prompt in the admin panel. The goal is a warmer tone.
A trace is the story of one request. Without it, "the bot gave a wrong answer" is a complaint you cannot investigate.
The prompt registry stores named prompts with numbered, immutable versions. Think of it as the model registry from the MLflow module, but for text.
A change you cannot measure is a guess. The measurement is a golden set: a fixed list of questions with what a good answer must contain, plus scorers...
mlflow.genai.evaluate runs your app over the golden set, applies the scorers and logs everything to a run: the metrics, one trace per example, and...
A gate is a script whose exit code stops a release.
On a Friday afternoon at acme-shop, someone edits the support chatbot's prompt in the admin panel.
A trace is the story of one request. Without it, "the bot gave a wrong answer" is a complaint you cannot investigate.
The prompt registry stores named prompts with numbered, immutable versions.
A change you cannot measure is a guess. The measurement is a golden set: a fixed list of questions with what a good...
mlflow.genai.evaluate runs your app over the golden set, applies the scorers and logs everything to a run: the metrics...
A gate is a script whose exit code stops a release.
A gate protects the moment of release. Production observability protects every minute after it.
Automated scores tell you whether answers look right by your own rules.
You can build this stack on other platforms.
You will build a small retrieval chatbot for acme-shop support, trace it with personal data removed before storage...
Quick reference Common mistakes The first mistake is editing prompts in place.
Roles that use the skills in this module.
AI Engineer
MLOps Engineer
Platform Engineer
Site Reliability Engineer
Not a software engineer sheet. Every problem comes from real DevOps, SRE, Platform and Cloud interviews, from your first script to a system you build yourself.
Open the Coding SheetGit records who changed the text and why, but your running service needs a stable name to load and a way to move that name back in seconds. A registry gives immutable numbered versions, aliases such as production, and tags that hold the gate evidence. Many teams keep the text in Git and register each approved change, so they get both.
Not safely. A judge is itself a model, so its scores are noisy and can drift when the judge changes. Use cheap deterministic scorers for the things you can check with code, such as citations and key facts, and add judge scores as a second opinion with a wide enough margin and repeated runs.
Before it is stored, not after. A trace keeps the inputs and outputs of every step, so an email typed into the chat box is stored the moment the request is traced. Mask it at the application entry point and again in a span processor that runs before storage, then limit who can read traces and how long they are kept.
No. MLflow Tracing is built for LLM-specific questions: what was retrieved, what was sent to the model, how many tokens it used and what it cost. OpenTelemetry with Jaeger is better for the whole request path across many services. MLflow can export spans to an OpenTelemetry endpoint, so you can use both.
Large enough that one example is not a big share of the score. With 14 examples, one failure moves a metric by about 7 points, which is useful for a lab and thin for production. Start with the questions your users ask most and the ones that have hurt you, grow it from reviewed failures, and keep part of it as a holdout.