Feature Store with Feast for Training and Serving

This milestone creates the standalone repository, a pinned Python environment and a Redis container that Feast will use as its online store.

The project lives in its own repository, separate from mlops-lab, so it reads as a standalone piece of work on GitHub.

Step 1

Create the repository layout, a virtual environment and the pinned requirements.

Save requirements.txt:

Save pyproject.toml, which makes featurekit an installable package and tells pytest where the source lives:

Note

feast[redis] installs Feast plus the Redis client it needs; plain feast cannot talk to Redis. If pip reports a version conflict between Feast and the FastAPI or Uvicorn pins, remove those two pins, let pip choose versions Feast accepts, and record what it chose in requirements.lock. The lock file is what a reviewer installs, so commit it.

Add a .gitignore containing .venv/, data/, models/ and __pycache__/. Keep results/ tracked: the JSON files written there are the evidence your README quotes. Generated data and models are rebuilt by the pipeline, never committed.

Step 2

Start Redis in Docker, bound to your laptop only. Save docker-compose.yml:

Security

This Redis has no password, which is acceptable only because the port is bound to 127.0.0.1. Never publish it as 6379:6379, which listens on every interface. In production, the Feast module's production topic covers TLS, a password from a secret store, and a rendered config file that never enters Git.

Note

--save "" and --appendonly no turn off persistence. The online store is a cache of the offline store: if Redis loses its data, you rebuild it with one materialization run. That property is worth stating in your README, because it is why Redis is safe to treat as disposable here.

Step 3

Point Feast at Redis. Save feature_repo/feature_store.yaml:

The registry path is relative to feature_repo/, so the registry lands in feature_repo/data/. Add that folder to .gitignore too: the registry is rebuilt by feast apply from the definitions, which are what you commit.

Common Mistake

Installing plain feast instead of feast[redis]. Everything works until the first feast apply or materialization against Redis, which then fails with a missing Redis client. Install the extra from the start and record it in the lock file.

This milestone writes down exactly what every feature means, in a contract file, a shared Python module and tests that fail until the pipeline matches the contract.

Before any pipeline code, write down what each feature means. The counter incident happened because two teams had one feature name and two meanings. A contract is the one place the meaning lives, and tests make it executable.

Step 4

Write FEATURES.md at the repository root. This is the human-readable contract a reviewer reads first.

Note

The windows are half-open, written [h-24h, h): they include the start and exclude the end. Half-open windows are the usual choice for time buckets, because consecutive windows never overlap and never leave a gap. The exact choice matters less than writing it down and testing it.

Step 5

Turn the contract into code that training, serving and tests all import. Save src/featurekit/contract.py:

Step 6

Write the definition tests now, before the pipeline exists. They will fail until Step 8, which is the point: the tests describe the contract, and the pipeline is written to satisfy them. Save tests/test_definitions.py:

Note

Each test builds a tiny history by hand where you can work out the right answer on paper. That is what makes it an independent check: the expected value comes from the contract, not from running the pipeline and copying its output.

Common Mistake

Writing the expected value in a test by running the pipeline and copying its output. That only proves the code agrees with itself. Work out each expected value on paper from FEATURES.md, then let the code prove it matches.

This milestone generates seeded raw payments and turns them into hourly account and merchant snapshots that satisfy the definition tests.

The Feast module's lab looped over accounts and days. That is fine for 30,000 rows and far too slow for hourly data. This pipeline buckets payments by hour once, then uses rolling sums, which pandas runs in compiled code.

Step 7

Write the seeded data generator. Save src/featurekit/generate.py:

Note

The data ends one hour before the current hour, and every timestamp is timezone-aware UTC because now is. The data is synthetic: risky accounts and compromised merchants are hidden flags the model never sees directly, only through the features.

Step 8

Write the feature pipeline. Save src/featurekit/pipeline.py:

Note

shift(skip) moves every bucket skip hours later before the rolling sum, so the value at hour h only sums buckets that ended before h. With skip=1 that is the "strictly before h" rule; with skip=73 it is the chargeback delay. A single wrong shift is exactly the off-by-one from the counter incident, which is why the definition tests check both.

Step 9

Run the tests, then the pipeline.

Expected output (the payment count and rate depend on library versions; the shape should match):

300 accounts times 720 hours gives 216,000 account rows, and 40 merchants times 720 hours gives 28,800. If your counts differ slightly, check how many whole hours your run covered; the generator anchors on the current hour.

Common Mistake

Mixing timezone-naive and UTC timestamps between the generator, the pipeline and the entity dataframe. Feast then compares times that are hours apart, and point-in-time joins return empty features with no error. Keep every timestamp timezone-aware UTC end to end.

This milestone registers two entities, two feature views and one feature service that spans both, so training and serving ask for the same features by one name.

Now register the tables with Feast. The new part compared with the module is a second entity: one feature service now pulls features keyed by two different IDs.

Step 10

Save feature_repo/features.py:

Note

The contract tag links each feature view to the version of FEATURES.md it implements. When the contract changes meaning, bump it, and create a new feature service name, so a model trained on v2 never silently reads v3 features. That naming discipline is a team convention; Feast does not enforce it.

Step 11

Register the definitions.

Expected output, similar to:

Why one service can span two entities

A feature service is a list of features, not a table, so its features can belong to different entities. At retrieval time you pass every join key the service needs: the training entity dataframe has both account_id and merchant_id columns, and each online request passes both IDs. Feast looks up each feature view with its own key and returns one combined row. Forget one key and Feast raises an error instead of guessing, which is the behaviour you want.

Common Mistake

Changing what a feature means while keeping its name and the feature service name. The model in production keeps reading the new values as if they were the old ones. Bump the contract version and create a new feature service, such as fraud_v3, for any change in meaning.

This milestone builds the training set through Feast, checks every feature value against an independent pandas as-of join, and trains a model with an honest, time-based evaluation.

This is where the project proves its most important claim: no training row saw information from after its own timestamp. Feast builds the training set, and an independent pandas join checks every value Feast returned.

Step 12

Save src/featurekit/train.py. It builds the training set through Feast, verifies all five features against merge_asof, applies the contract defaults, drops a warm-up period, trains on a time split, and compares with an intentionally invalid "latest stats" join. The final test window is only scored when you pass --final, and only once.

Note

A bootstrap interval resamples the validation rows with replacement a few hundred times and recomputes PR-AUC each time. The middle 95 percent of those values shows how much the score would move with a different sample of the same size. With a few dozen fraud cases in validation, that range is wide, and reporting it is more honest than a single number with three decimals.

Step 13

Run it.

Illustrative output (your numbers will differ; the shape should match):

Read the result in three parts. Zero differing values means Feast's join and your independent join agree on every one of roughly 22,000 feature values, so the training set respects both event and available times. The point-in-time PR-AUC comes with an interval about a quarter wide, so a change of 0.02 between two runs means nothing. And the invalid latest join scores far higher because it reads chargebacks and merchant rates from the end of the period, including ones caused by the validation payments themselves. It is a deliberately broken baseline that shows the size of the leak; it is never a model you would ship.

Common Mistake

Choosing features, thresholds or model settings by looking at the final test window. Every decision in this project uses validation rows. The test window is scored once, in Step 24, and the script refuses to score it twice.

Why the warm-up period is dropped

The data starts on day 0, so on day 2 a 7-day average has only 2 days of history and a 90-day chargeback count has almost none. Those early rows describe a world where every account looks new, which never happens in production. Dropping the first 7 days removes the worst of that. The 90-day chargeback window is still never full in 30 days of data; say so in your README as a known limit of the synthetic setup rather than hiding it.

This milestone loads the latest feature values into Redis and proves, entity by entity, that Redis holds what the offline store considers current.

The training side is proven. Now load the online store and prove it holds the same values the offline store considers current.

Step 14

Save src/featurekit/materialize.py:

Tip

This project materializes a fixed 3-day window every time instead of using materialize_incremental. Rewriting a few hundred thousand rows costs seconds, and a fixed window is idempotent: rerunning it after any failure always produces the same online state. At millions of entities, incremental runs become worth their extra state to track.

340 keys: one per account plus one per merchant, each holding that entity's latest feature values. The exact key layout is Feast's internal format; never read or write it directly.

Step 15

Save src/featurekit/parity.py, the offline-to-online materialization parity check. It pairs every account with a merchant so that every entity of both kinds is checked, and compares Feast's offline answer "as of the newest snapshot" with Redis.

Note

The comparison time is the newest snapshot time plus one second, not the wall clock. Comparing "as of now" would fail on a slow afternoon for the wrong reason: once the newest snapshot is older than the 3-hour TTL, the offline join returns nothing while Redis still returns the last values. Staleness is real, but it is the freshness check's job, not this one's.

Common Mistake

Treating a passing parity check as proof the features are correct. Parity compares two copies of the same numbers; if the pipeline computed them wrongly, both copies agree. Meaning is proven by the definition tests from Milestone 2.

This milestone adds a FastAPI scoring endpoint that reads account and merchant features from Redis at request time and fills gaps with the same defaults training used.

The online store is only useful if a prediction can read it fast, with the same defaults training used. This endpoint is a slimmed-down version of fraud-api from Model Serving with FastAPI and BentoML, with the feature lookup added.

Step 16

Save src/featurekit/api.py:

Note

lookup_ms is returned only so this project can measure the feature lookup on its own. In a real service you would record it as a metric, not send it to checkout. The defaults come from contract.py, the same dictionary train.py passed to fillna, so a new account is represented identically in training and serving.

Step 17

Measure the lookup. Save src/featurekit/bench.py, which calls the API in-process 500 times with random known IDs and reports server-side lookup percentiles:

Illustrative output from a laptop:

This is one client calling sequentially against local Redis, so it measures the cost of one lookup, not capacity under load. Put the numbers in your README with that sentence next to them. The Model Serving module's Locust setup is the tool for a concurrency test.

Common Mistake

Creating the FeatureStore object inside the request handler. Every request then re-reads the registry and opens a new connection, which can cost more than the lookup itself. Create it once in the lifespan handler, as above.

This milestone adds a two-part freshness check, then breaks the system three ways to show that definition tests, freshness and parity each catch a failure the others miss.

Each drill below breaks one thing, and each is caught by a different check. That is the argument for having all three.

Step 18

Save src/featurekit/freshness.py, the two-part freshness check from the Feast module, extended to both views and to entity coverage:

If data_fresh is false, more than 90 minutes have passed since you ran the pipeline. That is the check working; rerun python -m featurekit.pipeline and python -m featurekit.materialize.

Step 19

Drill 1, the counter bug. In pipeline.py, change the first line of account_snapshots that computes acct_txn_24h from window(agg, "n", 24, 1) to window(agg, "n", 24, 0), which lets each hour count payments in its own bucket. Run the tests.

That one-character change is the acme-shop incident: the count now includes the payment being scored. Parity would not catch it, because both stores would hold the same wrong number. Only a test of the definition can. Change it back to 1 and confirm all four tests pass.

Step 20

Drill 2, the half-finished job. Simulate an hourly run that crashed after writing 20 accounts:

Illustrative output (trimmed):

Parity passes, because Redis faithfully holds what the offline store contains. Materialization looks healthy, and the newest data is fresh. Only the coverage check notices that 280 accounts did not get this hour's snapshot. Rebuild with python -m featurekit.pipeline followed by python -m featurekit.materialize.

Step 21

Drill 3, the lost cache. Restart Redis; with persistence off, it comes back empty.

This time freshness reports healthy, because the registry still records the last materialization and the source data is fine. Parity catches it. Every request to the API would now get all-default features with five warnings, which a dashboard on the warning rate would also show within a minute. Recover with one command, python -m featurekit.materialize, and confirm parity is back to 300/300.

Common Mistake

Keeping only one of the three checks because it "usually catches things". Each drill above passed at least one check. Run all three on every scheduled cycle.

This milestone turns the working code into a repository a reviewer can run with one command and understand in five minutes.

A reviewer gives a portfolio repository about five minutes. In that time they should be able to run one command, see every proof pass, and understand what you would do next.

Step 22

Save a Makefile so the whole project runs from one command.

Note

The target: ; command form puts a one-line recipe on the same line as the target, so no tab characters are needed. make all runs every proof in order and stops at the first failure, which is exactly what a reviewer wants to see.

Step 23

Write README.md with these sections, in this order. Keep each one short; link to FEATURES.md instead of repeating it.

• Problem: two sentences on training-serving skew and leakage in fraud features, with the counter incident as the example.
• Architecture: the Mermaid diagram from this project, which GitHub renders.
• Quick start: python3.12 -m venv .venv, install from requirements.lock, make all.
• Proofs: the acceptance table from the first topic, with the command for each line.
• Results: validation and final PR-AUC with intervals, the invalid-join score labelled as a leak demonstration, and lookup p50 and p95 with the "one sequential client" caveat.
• Drills: the three-row table from Step 21.
• Limits: synthetic data, hourly batch features rather than a live counter, a 90-day window that is never full, single-node Redis without TLS.
• Next steps: two or three items from the stretch goals below.

Commit, then tag the release so a reviewer sees a finished version: git tag v1.0.

How to talk about this project in an interview

Interviewers ask "walk me through a project". Use this order, and keep it under three minutes.

1. The problem: a fraud model that looked good offline and failed live because a feature meant different things in training and serving.
2. The design: one feature contract, hourly features from one pipeline, Feast with Parquet offline and Redis online, two entities in one feature service.
3. The proofs: definition tests for meaning, an independent as-of join for leakage, parity and two-part freshness for the online store.
4. A failure you caught: pick one drill and say which check caught it and why the others could not.
5. The numbers, honestly: PR-AUC with its interval, and why the invalid join's higher score is a warning, not a result.

Stretch goals

Each of these extends the project in a direction real teams take. Pick one or two; do not try all of them.

• Add a streaming-style counter: a Redis INCR per payment for acct_txn_24h, and run the same definition tests against it. This is the acme-shop incident built properly.
• Schedule pipeline, materialize, parity and freshness hourly with an Airflow DAG, using the patterns from Automating ML Training Pipelines with Airflow.
• Run make test and a reduced make all in GitHub Actions on every pull request; CI/CD and Continuous Training for ML covers the gate design.
• Log the training run, the Feast feature service name and the contract version to MLflow, so a registered model records which features it needs.

Common Mistake

Listing tools in the README instead of proofs. "Feast, Redis, FastAPI" tells a reviewer nothing they could not guess from the file names. The acceptance table, results with intervals and the drills table show what the system guarantees.

This step runs every proof from a clean start, scores the reserved test window once, and shuts the project down cleanly.

Run everything from scratch

Delete the generated state and rebuild it with the Makefile, exactly as a reviewer would. Every target must pass.

Expected output, trimmed to the key lines (numbers illustrative):

Validation checklist

Check the API's default handling by hand once:

An unknown account and merchant come back with five warnings and a score built from the contract defaults, the same values the training set used for entities with no history.

Step 24

Score the final test window, once. Do this only after every decision in the project is made.

Illustrative output:

Report this number in your README next to the validation result. If they differ by less than the width of the intervals, that is noise, not a finding. Running it again stops with "final test already scored", which is the guard working.

Step 25

Shut the project down. Redis holds no data you cannot rebuild, so removing it is safe.

Commit results/validation.json and results/final_test.json with the code: they are the evidence your README quotes.

Quick reference

Common mistakes

Writing the pipeline before the contract is how one feature name ends up with two meanings. It happens because code feels like progress and documentation does not. The fix is the order this project used: write FEATURES.md, encode its rules in contract.py and definition tests, then write the pipeline until the tests pass.

Trusting parity as proof of correctness misses every bug that lives upstream of the stores. Parity compares Redis with Parquet; if the pipeline computed the wrong number, both hold the same wrong number and parity passes, as Drill 1 showed. The fix is to keep definition tests for meaning and parity for materialization, and never let one stand in for the other.

Comparing offline and online "as of now" makes the parity check fail whenever the data is older than the TTL, so people learn to ignore it. It happens because "now" feels like the natural reference time. The fix is to compare as of the newest snapshot, and leave staleness to the freshness check, which has its own threshold.

Checking only when materialization last ran reports healthy while the upstream job writes partial or no data, as Drill 2 showed. The fix is to check the newest event time and the number of entities it covers, alongside the materialization time.

Letting training see NaN while serving sends zeros creates skew inside your own code. It happens because tree models accept missing values without complaint. The fix is one DEFAULTS dictionary in contract.py, applied with fillna in training and in the API.

Reporting the invalid latest-join score, or a single PR-AUC without its interval, overstates what the model can do. It happens because higher and more precise numbers look better in a README. The fix is to report the point-in-time score with its bootstrap interval, label the leaky score as a demonstration, and score the test window once.

Publishing Redis on every interface because the default Docker port mapping does it is a quiet security hole on any shared network. The fix is to bind to 127.0.0.1 locally, and in production to use TLS, a password from a secret store and network rules that admit only the materialization job and the serving pods.

What you built and what comes next

You built acme-fraud-feature-store: a written feature contract with tests, an hourly vectorized feature pipeline for two entities, a Feast repository with Parquet and Redis, a point-in-time training set verified value by value against an independent join, a model with an honest score and interval, a FastAPI endpoint reading features at request time with shared defaults, and three checks that each caught a different failure. It runs from one command and explains itself in its README.

With this project, Step 5 is complete. Step 6 continues with Batch Inference and Model Optimization, and Step 7's CI/CD and Continuous Training for ML is where make test, make parity and the final-score guard become gates that run on every change.