Model Serving with FastAPI and BentoML

Start with FastAPI when you have one model, modest traffic and a team that already runs Python web services, because you control every line. Move to BentoML when you need adaptive batching, several models in one service or a standard packaging format that builds images for you. Both can load the same MLflow registry model, so the choice is reversible.

An alias such as champion lets the team promote or roll back a model by moving a pointer in the registry, without editing code or config in the service. The service should still resolve the alias to a fixed version once per load and report that version in every response. That way you get easy promotion and a precise record of which model made each decision.

Start with one worker, then load test while you watch p95 latency and memory. Add workers only while latency improves and memory allows, because every worker holds its own copy of the model and native libraries such as XGBoost start threads of their own. The right number depends on the CPU you allocate, the model size and the traffic you need to carry.

No. The test set is looked at once, at the end of model development, and serving checks run many times. Use a small sample of validation rows to prove the served model returns the same scores as the offline model. That parity check tests the serving code, not the model's quality.