Every module before this one in the roadmap was something every data engineer needs to complete. This one is different - it is not a list of technologies you must finish, it is a map of directions you can explore. A data engineer at Razorpay building fraud pipelines and a data engineer at Meesho building the feature store for a recommendation model are both excellent, both employable, and both spend their advanced hours on almost none of the same tools. * Everything before this step - SQL, Python, pipelines, warehousing, streaming, data quality - is what makes you a working data engineer. This module is about what makes you a data engineer with a direction. * The hierarchy here has three levels, worth being explicit about upfront so it is clear exactly what you are choosing between: ```text Three primary specialisations (covered in depth): 1. Analytics Engineering 2. Platform Data Engineering 3. AI/ML Data Infrastructure - Path A: ML Data Infrastructure - Path B: AI/LLM Data Infrastructure One additional technical direction (covered more briefly, since the core skills were already taught earlier in this roadmap): - Streaming and Real-Time Engineering One architectural model to be aware of, not a specialisation you personally choose: - Data Mesh ``` * Pick one of the primary specialisations, or the streaming direction, based on what kind of problems you actually enjoy solving - not based on which sounds most impressive on LinkedIn. * Each specialisation gets a working map of the territory, not a full mastery course - depth comes later, through the job you take and the projects you choose once you know which direction you are headed.
Every data engineer starts as a generalist - that is exactly what the rest of this roadmap has been building. **Specialisation** means choosing one area to go noticeably deeper in than the rest, not abandoning everything else you have learned. A specialised data engineer still writes SQL, still builds pipelines, still thinks about data quality - they just also happen to be the person their team turns to first for one particular kind of hard problem. > 📌 **Engineering Decision:** pick a specialisation by asking yourself one honest question - when a project went well, what part of it did you enjoy most? If it was making messy business logic finally make sense in clean SQL, look at Analytics Engineering. If it was building the tool that let five other teams stop asking you for help, look at Platform Data Engineering. If it was the moment a model actually worked because the data feeding it was solid, look at AI/ML Infrastructure. If none of those quite fit yet, that is a completely reasonable place to be - stay a strong generalist a while longer and revisit this module in six months.
Analytics engineering is the discipline of turning raw pipeline output into business-trustworthy metrics - the layer between "the data landed in the warehouse" and "the CFO trusts this number in a board deck." You already met the fundamentals of this world in the dbt module; this specialisation is what a career built specifically around that skillset looks like. ### Advanced dbt patterns * **Custom macros** - reusable Jinja-templated SQL snippets that remove repetition across models, the same instinct as writing a Python function instead of copy-pasting a block five times * **Packages** - shared libraries of macros and models that other teams have already built and tested, installed into a dbt project the way a Python library is installed with pip * **Cross-database compatibility** - writing dbt macros that produce correct SQL whether the underlying warehouse is Snowflake, BigQuery, or Redshift, useful at companies that run more than one warehouse or are migrating between them ```sql -- A simple custom macro - Jinja templating inside SQL {% macro rupees_to_lakhs(column_name) %} ROUND({{ column_name }} / 100000.0, 2) {% endmacro %} -- Used inside a model like a function call SELECT restaurant_id, {{ rupees_to_lakhs('total_revenue') }} AS revenue_in_lakhs FROM {{ ref('fct_daily_orders') }} ``` > **Note:** `{{ ref('fct_daily_orders') }}` and the macro call above both use Jinja, dbt's templating language - the double curly braces tell dbt "compute this before sending real SQL to the warehouse." A macro is essentially a SQL-generating function. ### Semantic modeling and metric governance - the discipline behind the semantic layer A **semantic layer** is a single, centrally defined place where business metrics like "revenue" or "active user" are calculated once, so that a dashboard, a data scientist's notebook, and a finance report never quietly disagree with each other because someone wrote `SUM(amount)` slightly differently in three places. But defining a metric well requires more than a semantic layer tool - it requires **semantic modeling**, the discipline of deciding, in plain business language before any SQL is written, exactly what a metric means. Consider how many different things "revenue" could mean at a company like Swiggy: Revenue | ├── Gross Revenue (every order placed, cancelled or not) ├── Net Revenue (gross minus refunds and cancellations) ├── Delivered Revenue (only orders that actually completed delivery) └── Recognised Revenue (revenue counted per accounting rules, which may differ from the date the order happened) All four of these could technically be called "revenue" by someone in a meeting, and each produces a genuinely different number from the same underlying orders table. Semantic modeling is answering, explicitly and in writing, questions like: what does one row represent, what business events should and should not count toward this metric, and who is the accountable owner if the definition ever needs to change. Only once those questions have real answers does building the metric in a semantic layer tool actually solve anything - the tool enforces a decision, it does not make the decision for you. > 📌 **Remember:** the problem a semantic layer and semantic modeling solve is not a technical one - it is a trust one. The moment two teams present different numbers for "monthly revenue" in the same meeting, every number in every future meeting gets questioned. Getting the underlying business definition right, and then enforcing it in exactly one place, is what makes that scenario structurally impossible. ### Building data products Analytics engineers increasingly treat a well-built mart table the way a software engineer treats a public API - it has an owner, a defined contract (which columns exist, what grain each row represents), a documented SLA for freshness, and known consumers who would be told before a breaking change ships. This is the mindset shift from "I made a table" to "I own a data product other teams depend on." > **Note:** what does this actually look like week to week? A typical week for an analytics engineer involves reviewing and merging dbt model changes, tracking down why two dashboards disagree on a metric that should be identical, defining a new metric with stakeholders before writing any SQL for it, and keeping test coverage healthy as the mart layer grows.
Platform data engineering is not about building any single pipeline - it is about building the infrastructure and tooling that lets every *other* engineer at the company build pipelines faster and more safely, without filing a ticket and waiting on you personally. ### Self-service data infrastructure Picture a mid-sized team at Flipkart where every new data source requires a data engineer to hand-write a new Airflow DAG. That does not scale past a handful of engineers. A platform data engineer instead builds a **template or internal tool** - for example, a YAML file where any engineer declares "pull this API, land it in this S3 zone, on this schedule" and the platform generates and deploys the DAG automatically. Old way: Platform way: engineer -> files ticket engineer -> writes 10-line YAML -> waits days -> pipeline deploys itself -> DE builds DAG by hand -> DE only reviews, doesn't build ### Platform abstractions and developer experience The real mindset behind good platform engineering is captured in one question: how do I turn the *correct* way of doing something into the *easiest* way of doing it? This is where terms like **golden paths** come from - a pre-approved, pre-tested way of building a pipeline that is so much less effort than doing it manually that engineers choose it naturally, not because a policy forces them to. * **Pipeline templates** - pre-built, parameterised DAG patterns for common cases (API ingestion, file drops, database CDC) so nobody starts from a blank file * **Internal CLIs and SDKs** - command-line tools or small libraries that wrap common platform operations, the same way a company might build an internal `company-cli deploy` command instead of expecting every engineer to remember raw deployment steps * **Infrastructure modules** - reusable Terraform modules for common data infrastructure patterns, so provisioning a new S3-backed data source follows one proven pattern instead of being hand-written each time * **Standardised deployment** - one well-tested way pipelines get promoted from dev to production, instead of every team inventing its own release process A platform engineer also naturally ends up caring about metadata, catalog, and lineage - not because governance is their job title, but because a self-service platform that lets anyone spin up a new pipeline also needs to automatically know who owns it, what it produces, and what depends on it downstream, or the platform itself becomes ungoverned chaos at scale. Pipeline | v Dataset produced | v Owner assigned automatically | v Schema registered | v Lineage connected | v Downstream consumers tracked ### Data platform observability This is a different kind of monitoring than the data quality checks covered earlier in this roadmap. Data quality asks "is this specific table's data correct?" Platform observability asks "is the platform itself healthy?" - are DAGs finishing on time across the board, is Airflow's scheduler under strain, are storage costs trending in a direction someone should worry about. ### Cost management for data platforms at scale At scale, a data platform's AWS or Snowflake bill becomes a real budget line, and a platform engineer is often the person who owns bringing it down - right-sizing compute, enforcing S3 lifecycle policies across every team's bucket, catching a runaway query before it becomes a five-figure surprise on next month's invoice. > **Note:** a typical week for a platform data engineer involves building or improving a pipeline template, fielding questions from other engineers using the self-service platform, investigating why the Airflow scheduler is under strain, and reviewing infrastructure cost trends before they become a problem someone escalates.
Every model a data scientist trains and every AI feature a product ships is downstream of a data engineer who built the pipeline feeding it. This specialisation actually contains two related but genuinely different paths, worth understanding as distinct before picking one - someone who loves feature engineering pipelines may have very little interest in vector search, and vice versa. ### Path A: ML Data Infrastructure - feature stores and training data A **feature** is a computed input to a machine learning model - "this customer's average order value over the last 30 days" is a feature, not a raw column. Without a feature store, the same feature often gets computed twice: once by a data scientist in a training notebook, and again, slightly differently, by an engineer building the live prediction service. A **feature store** - Feast is a widely used open-source option, and a useful choice for learning the concepts - is a system designed to centralise feature definitions and reduce this kind of training-serving mismatch by making the same feature logic reusable across both training and inference. * **Offline store** - features computed in bulk over historical data, used to train a model * **Online store** - the same features, kept fresh and served with millisecond latency, used when a live prediction request comes in * **Training-serving skew** - the general term for what happens when the offline and online paths disagree, even slightly, about how a feature is calculated - a feature store reduces this risk significantly, but it does not eliminate it automatically. Stale online features, a materialisation job that silently failed, or a point-in-time calculation error can all still cause skew even with a feature store in place - the tool centralises the *definition*, it does not guarantee every pipeline feeding it is perfectly reliable. > 📌 **Engineering Decision:** reach for a feature store only once a company has more than one model reusing the same features, or once training-serving mismatches have actually caused a production bug. For a single model with a handful of features, a feature store is real infrastructure overhead that is not yet paying for itself - a well-organized dbt mart table is often enough at that stage. > **Note:** a typical week for someone on this path involves building or maintaining a feature pipeline, investigating why an online feature looked stale during a live prediction, working with a data scientist to add a new feature to the offline store, and monitoring feature freshness as part of the platform's overall health. ### Path B: AI and LLM Data Infrastructure - preparing text for Retrieval Augmented Generation **RAG (Retrieval Augmented Generation)** lets an AI system answer questions using a company's own documents instead of only what a model learned during training. Much of the pipeline behind a production RAG system is genuinely data engineering work - ingestion, parsing, transformation, metadata management, indexing, and keeping the knowledge base synchronised as source documents change. Depending on the company, the embedding and retrieval architecture itself may be owned solely by data engineering, or jointly with ML or AI engineers - smaller companies tend to have one person or team own the whole pipeline end to end, while larger ones split ingestion (data engineering) from retrieval architecture (often ML or AI engineering) more formally. Ingestion -> Parsing -> Cleaning -> Chunking -> Metadata enrichment -> Embedding -> Indexing -> Retrieval -> Monitoring/re-indexing * **Chunking** - splitting long documents into smaller pieces, since a model can only usefully search and reason over chunks of a manageable size, not an entire 200-page policy document at once * **Metadata enrichment** - tagging each chunk with information beyond the raw text itself: which document it came from, when that document was last updated, which team or product it relates to, and critically, who is allowed to see it. Without metadata, a RAG system is just "text in, text out" - metadata is what makes results filterable, permission-aware, and traceable back to a source, and skipping it is one of the most common gaps in a first RAG build. * **Embedding** - converting each chunk of text into a list of numbers (a vector) that captures its meaning, so that pieces of text with similar meaning end up mathematically close to each other * **Vector databases** - Pinecone, Weaviate, and pgvector (a PostgreSQL extension) are purpose-built to store these vectors and quickly find the ones closest in meaning to a new query * **Re-indexing** - source documents change, get updated, or get deleted, and a production RAG pipeline needs an ongoing process to detect that and refresh the corresponding vectors, rather than serving stale or deleted content forever > **Note:** this is the same "chunking" and "embedding" language used across nearly every RAG tutorial, and it can sound more exotic than it is. Chunking is closer to writing a smart `split()` function than anything mathematically deep, and picking an embedding model is closer to picking a library than building one - the genuinely hard data engineering work is the pipeline reliability around it: keeping the vector database in sync as source documents change, handling documents that fail to parse, enforcing the access permissions carried in a chunk's metadata at query time, and re-embedding efficiently when the underlying model gets upgraded. > **Note:** a typical week for someone on this path involves debugging why a document failed to parse during ingestion, tuning chunk size and overlap after noticing retrieval quality issues, adding metadata fields so results can be filtered by permission or product line, and building or improving the re-indexing job that keeps the vector store in sync with changing source documents.
You already covered the core technical ground for this direction back in the Streaming Data Engineering step - Kafka topics and partitions, consumer groups, Kafka Connect, Schema Registry, and a working understanding of Flink SQL. Streaming as a specialisation means going noticeably deeper into that same territory rather than learning new tools from scratch. Someone specialising here spends their advanced hours on problems like: designing event contracts and schema evolution rules that will not break consumers months later, reasoning carefully about event-time versus processing-time and how watermarks handle data that legitimately arrives late, choosing between at-least-once and exactly-once delivery semantics and understanding what each actually costs in complexity and infrastructure, building and operating stateful stream processing jobs that must survive restarts without reprocessing or losing data, and building real-time observability specifically for streaming systems, where "the pipeline succeeded" is a much fuzzier concept than in batch. > **Note:** a typical week for a streaming specialist involves debugging consumer lag that crept up overnight, reviewing a proposed schema change for backward compatibility before it ships, tuning a Flink job's checkpoint interval after an incident, and working with a product team to define what "real-time" actually needs to mean for a new feature - often the honest answer is "a 30-second batch would be fine," which is itself a valuable thing for a streaming specialist to know how to say. If Kafka and Flink were your favourite part of this roadmap, treat this as a genuine fourth option alongside the three specialisations above - not a lesser one.
Every module before this one in the roadmap was something every data engineer needs to complete. This one is different -...
Every data engineer starts as a generalist - that is exactly what the rest of this roadmap has been building. Specialisa...
Analytics engineering is the discipline of turning raw pipeline output into business-trustworthy metrics - the layer bet...
Platform data engineering is not about building any single pipeline - it is about building the infrastructure and toolin...
Every model a data scientist trains and every AI feature a product ships is downstream of a data engineer who built the ...
You already covered the core technical ground for this direction back in the Streaming Data Engineering step - Kafka top...
Data mesh deserves a different label than the specialisations above it, because it is not something you personally "choo...
There is no wrong answer among these, and the honest truth is that plenty of working data engineers stay strong generali...
Pick exactly one of the exercises below, matching whichever specialisation interests you most. You do not need to do all...
By the end of this module, you should be able to fill in something like the short worksheet below. Writing it down, even...
Specialisation Core Tools Best Fit For Analytics Engineering dbt, semantic layers, SQL Turning raw data into trusted, we...
This module does not carry a "5 common mistakes" section the way earlier modules do, because it is not teaching a skill ...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.