Learn batch scoring for ML: chunked jobs, safe output writes, latency and throughput measurement, ONNX export, model shrinking and scheduled runs.
It is 6 AM at acme-shop. The risk team starts its day with a list: the payments from the last 24 hours that look most likely to be fraud, ranked from...
A batch job looks simple: read data, score it, write the results. Most real problems come from the edges: bad input, half-written output and reruns.
The single most useful thing you can do for model speed costs no money and no new tool: stop scoring one row at a time.
You cannot improve what you do not measure. Three numbers describe how a model performs, and each answers a different question.
So far the model has been a joblib file. It works, but it has two limits.
Once you have a .onnx file, ONNX Runtime runs it. The code is short.
It is 6 AM at acme-shop. The risk team starts its day with a list: the payments from the last 24 hours that look most...
A batch job looks simple: read data, score it, write the results.
The single most useful thing you can do for model speed costs no money and no new tool: stop scoring one row at a time.
You cannot improve what you do not measure.
So far the model has been a joblib file. It works, but it has two limits.
Once you have a .onnx file, ONNX Runtime runs it. The code is short.
A smaller model is cheaper to store, faster to load and, sometimes, faster to run.
A batch job is only useful if it runs when it should, and tells you when it does not.
You will turn the project into a batch job, break its input on purpose, compare ways to call the model, export the...
Quick reference Common mistakes Scoring one row at a time is the biggest and most common mistake.
Roles that use the skills in this module.
AI Engineer
MLOps Engineer
Platform Engineer
Not a software engineer sheet. Every problem comes from real DevOps, SRE, Platform and Cloud interviews, from your first script to a system you build yourself.
Open the Coding SheetUse batch scoring when nobody is waiting for the answer and the work can be done on a schedule, such as a nightly list of risky payments. Use real-time scoring when a decision is needed while a customer waits, such as approving a card payment. Many systems use both: real-time for the live decision, batch for reports, reviews and refreshes.
Each call to a model has a fixed cost: checking inputs, building arrays and starting the computation. Scoring one row pays that cost for one row, and scoring five thousand rows pays it once for all of them. That is why one-by-one scoring is slow, even when the model is small.
Not always. ONNX gives you a portable model file that does not need Python pickle and can run in many languages and environments, and the file is often smaller. For a small tree model that already runs in a batch, it may not be faster at all. Measure on your own model before you decide.
Write the output to a temporary file in the same folder, and rename it into place only when it is complete. Name the output after the day or run it belongs to, so a rerun replaces the same file and never adds to it. Readers then see either the old complete file or the new complete one.