It is 2 AM at Swiggy. A script that pulls yesterday's order data into the warehouse has failed silently for three nights in a row. Nobody noticed because the script used `print()` statements that vanished the moment the terminal closed. The finance team's revenue dashboard has been quietly wrong since Tuesday. This is not a story about a bad script. It is the default outcome of writing Python the way a tutorial teaches it, then running it in production. Data engineering Python is a different discipline from web development Python. A Flask app crashing shows a user an error page. A pipeline script crashing at 2 AM shows nobody anything unless you built it to. * Python is the glue language of the data stack. It rarely does the heavy lifting itself - Spark, Snowflake, and DuckDB process the actual rows - but Python is what tells those systems what to do, connects them to each other, and handles everything that does not fit neatly into SQL. * Python wins over Java or Scala for most data engineering work for three reasons: a vast ecosystem of connector libraries for nearly any system, code that is fast to write and easy for a teammate to read six months later, and it is simply what most data teams are already hiring and building around. * Web development Python builds apps that respond to users. Data engineering Python builds pipelines that run unattended, on a schedule, and must be trustworthy when nobody is watching. > 📌 **Remember:** a pipeline script's real user is not a person clicking a button. It is a scheduler running it at 3 AM with nobody around to notice if it quietly does the wrong thing.
Before writing a single transformation, you need a project that runs the same way on your laptop, your teammate's laptop, and the server that eventually schedules it. A **virtual environment** is an isolated copy of Python and its installed packages, kept separate from your operating system's Python and from every other project on your machine. ```bash ## Check which Python version you have - data engineering tools generally expect 3.10+ python --version ## Create a virtual environment named .venv inside your project folder python -m venv .venv ## Activate it - your terminal prompt will change to show (.venv) source .venv/bin/activate ## macOS/Linux ## .venv\Scripts\activate ## Windows equivalent ## Install the libraries this module uses pip install pandas requests pyarrow pytest ## Freeze exact installed versions so teammates get the identical setup pip freeze > requirements.txt ``` > **Note:** `requirements.txt` is a plain text list of every package and exact version your project needs. A teammate clones your repo, runs `pip install -r requirements.txt`, and gets the exact same environment - no "it works on my machine" surprises. A typical pipeline project keeps source code, data, and output cleanly separated: data-pipeline/ |-- src/ | `-- pipeline.py |-- tests/ | `-- test_pipeline.py |-- data/ |-- output/ |-- requirements.txt `-- README.md > 💡 **Tip:** this module deliberately stops at `venv` and `requirements.txt`. Tools like Poetry, uv, and Docker solve the same problem with more power and more complexity - they are worth learning later, but `venv` is enough to build and run everything in this module.
Every pipeline, no matter how complex, is built from the same small set of Python primitives used correctly and consistently. ### Variables, data types, and why type matters more here than in a script you run once A **variable** is a named container that holds a value while your program runs. In pipeline code, getting the data type wrong is one of the most common sources of silent bugs - a `price` column read as a string will sort `"9"` after `"100"` instead of before it. ```python order_id = "ORD10293" # str - IDs are usually kept as strings, even if numeric amount = 1499.50 # float - money amounts, watch for precision issues item_count = 3 # int is_delivered = True # bool discount_code = None # None - Python's "no value" - very common in real data print(type(amount)) # <class 'float'> ``` > **Note:** `None` is Python's way of saying "this value does not exist." It shows up constantly in pipeline code because real-world data has missing fields - a customer who never entered a phone number, an order with no discount applied. ### Lists and dictionaries - the two structures every pipeline leans on A **list** is an ordered collection you loop over - think of it as a stack of index cards. A **dictionary** is a set of key-value pairs - think of it as a labeled filing cabinet where you look things up by name, not position. ```python # A list of order amounts - order matters, duplicates are fine order_amounts = [499.0, 1250.0, 89.0, 1250.0] # A dictionary representing one Swiggy order - keys describe what each value means order = { "order_id": "ORD10293", "restaurant_id": "RST4471", "amount": 499.0, "status": "delivered" } print(order["amount"]) # 499.0 - look up by key, not position ``` ### Loops and conditionals - the decision logic of every transformation ```python orders = [ {"order_id": "ORD001", "amount": 650.0, "status": "delivered"}, {"order_id": "ORD002", "amount": 220.0, "status": "cancelled"}, {"order_id": "ORD003", "amount": 1899.0, "status": "delivered"}, ] total_delivered_revenue = 0.0 for order in orders: if order["status"] == "delivered": total_delivered_revenue += order["amount"] else: continue # skip cancelled orders entirely print(total_delivered_revenue) # 2549.0 ``` > 🔴 **Common Mistake:** writing `if order["status"] == "delivered"` without first checking the key exists crashes with a `KeyError` the moment one row is missing that field. Use `order.get("status")` instead of `order["status"]` when you are not certain every row has the key - `.get()` returns `None` instead of crashing. ### Functions - writing pipeline steps you can reuse and test A function is a named, reusable block of logic. Every real pipeline is built from small functions chained together, not one giant script. ```python def calculate_delivered_revenue(orders: list[dict]) -> float: """ Sum the amount field for every order with status 'delivered'. Orders with any other status (cancelled, pending) are excluded. """ total = 0.0 for order in orders: if order.get("status") == "delivered": total += order.get("amount", 0.0) # default to 0.0 if amount is missing return total ``` > **Note:** `*args` and `**kwargs` let a function accept a flexible number of arguments. `*args` collects extra positional arguments into a tuple, `**kwargs` collects extra named arguments into a dictionary. You will see these most often when writing wrapper functions around library calls - for example, a retry wrapper that needs to pass whatever arguments the wrapped function expects through to it, without knowing in advance what those arguments are. ### File handling and context managers - the correct way to touch files ```python # WRONG - the file handle may never get closed if an error happens mid-read f = open("orders.csv") data = f.read() f.close() # CORRECT - the "with" block guarantees the file closes, even if an exception occurs with open("orders.csv") as f: data = f.read() ``` > 📌 **Remember:** `with open(...) as f:` is called a context manager. It guarantees cleanup happens even when something goes wrong inside the block - this matters enormously in a long-running pipeline process that opens hundreds of files a day. Forgetting this leaks file handles until the process crashes. ### Error handling - because production data is never clean ```python def parse_amount(raw_value: str) -> float: """ Convert a raw string amount (e.g. from a CSV) into a float. Returns None if the value cannot be parsed, instead of crashing the pipeline. """ try: return float(raw_value) except (ValueError, TypeError): # ValueError: the string isn't a valid number, e.g. "N/A" # TypeError: raw_value was None, not a string at all return None finally: pass # finally always runs - useful for cleanup like closing a connection ``` > 🔴 **Common Mistake:** wrapping an entire pipeline function in one giant `try/except Exception` and printing "something went wrong" swallows the real error and makes debugging at 2 AM nearly impossible. Catch specific exceptions close to where they can actually happen, and always log the real exception message. > 💡 **Practice:** write three more functions using this pattern - one that counts orders per status, one that finds the single highest-value order in a list, one that returns only orders above a given amount passed in as a parameter. Run each against a small hand-made list first so you can check the answer yourself before trusting the code.
### Reading and writing CSV and JSON without pandas ```python import csv import json # Reading a CSV file row by row - each row becomes a dictionary with open("swiggy_orders.csv") as f: reader = csv.DictReader(f) for row in reader: print(row["order_id"], row["amount"]) # Writing JSON output summary = {"restaurant_id": "RST4471", "total_revenue": 45890.50} with open("summary.json", "w") as f: json.dump(summary, f, indent=2) ``` ### Environment variables - never hardcode a secret in pipeline code A **secret** is any value that should not be visible to anyone reading your code - a database password, an API key. Hardcoding it means it ends up in Git history forever, even if you delete it later. ```python import os # WRONG - this password is now permanently in your Git history DB_PASSWORD = "razorpay_prod_pass_2026" # CORRECT - read it from the environment, set outside the code DB_PASSWORD = os.environ["DB_PASSWORD"] # os.environ.get("DB_PASSWORD") returns None instead of crashing if it's unset - # use plain os.environ[...] when the pipeline should fail loudly if it's missing ``` > ⚠️ **Security:** a hardcoded secret committed to Git is not "fixed" by deleting the line in a later commit. The old commit still has it, and anyone with repo access can find it in the history. Rotate the credential immediately if this happens - do not just delete the line. ### datetime and pathlib - handling dates and paths correctly ```python from datetime import datetime, timedelta from pathlib import Path # Parsing a date string from a source system order_date = datetime.strptime("2026-08-14", "%Y-%m-%d") # Calculating "yesterday" for an incremental pipeline run yesterday = datetime.now() - timedelta(days=1) # pathlib handles path separators correctly across Linux, Mac, and Windows data_dir = Path("/home/data/swiggy") / "orders" / "2026-08-14.csv" print(data_dir.exists()) # True or False, without manual string concatenation ``` > 💡 **Practice:** write a function that reads a JSON file of restaurant metadata, and a second function that writes a filtered list of dictionaries back out as JSON with `indent=2`. Then write a script using `pathlib` that checks whether an expected input file exists before trying to open it, and logs a clear message instead of crashing if it does not.
**pandas** is a Python library built around the **DataFrame**, a table-shaped structure with rows and columns, similar to a spreadsheet or a SQL table loaded into memory. ```python import pandas as pd # Reading data - pandas can read CSV, JSON, and Parquet directly orders_df = pd.read_csv("swiggy_orders.csv") orders_df = pd.read_parquet("swiggy_orders.parquet") # requires pyarrow installed # Selecting and filtering delivered = orders_df[orders_df["status"] == "delivered"] high_value = orders_df[orders_df["amount"] > 1000] # Renaming columns orders_df = orders_df.rename(columns={"amt": "amount", "cust_id": "customer_id"}) ``` ### Cleaning and reshaping data - the operations every pipeline uses Real data arrives duplicated, unsorted, wrongly typed, and full of raw timestamp strings. These operations show up in nearly every transformation step you will write. ```python # drop_duplicates() - remove exact duplicate rows, or duplicates on specific columns orders_df = orders_df.drop_duplicates(subset=["order_id"], keep="last") # keep="last" keeps the most recent duplicate - useful when reprocessing overlapping data # astype() - explicitly cast a column's data type orders_df["amount"] = orders_df["amount"].astype(float) # pd.to_datetime() - parse raw timestamp strings into real datetime values orders_df["order_ts"] = pd.to_datetime(orders_df["order_ts"]) # once converted, you can do orders_df["order_ts"].dt.date or .dt.hour directly # sort_values() - order rows, most often by date for time-series style output orders_df = orders_df.sort_values("order_ts", ascending=False) # value_counts() - a fast way to see the distribution of a column, great for sanity checks print(orders_df["status"].value_counts()) # delivered 8420 # cancelled 312 # pending 44 # concat() - stack multiple DataFrames on top of each other, e.g. combining daily files all_days_df = pd.concat([monday_df, tuesday_df, wednesday_df], ignore_index=True) ``` > 🔴 **Common Mistake:** calling `pd.to_datetime()` without checking the result afterward - a handful of malformed date strings silently become `NaT` (pandas' null for dates) instead of raising an error. Always check `orders_df["order_ts"].isna().sum()` right after converting, so a bad source file does not quietly lose rows from every downstream aggregation. ### Reading large files without running out of memory `pd.read_csv()` loads the entire file into memory by default. A 10 GB file on a machine with 8 GB of RAM will crash the process before you write a single line of transformation logic. ```python # chunksize splits the file into pieces of 100,000 rows, processed one at a time total_revenue = 0.0 for chunk in pd.read_csv("large_orders.csv", chunksize=100_000): delivered_chunk = chunk[chunk["status"] == "delivered"] total_revenue += delivered_chunk["amount"].sum() print(total_revenue) ``` > **Note:** each `chunk` is a normal DataFrame, just a slice of the full file - every pandas operation you already know works the same way inside the loop. This is a stepping stone, not the final answer - when files stop fitting on one machine at all, that is what Spark (covered later in this roadmap) is built to solve. ### Handling nulls - the single most tested pandas skill ```python # isna() finds nulls, dropna() removes rows with nulls, fillna() fills them missing_report = orders_df["amount"].isna().sum() # count of null amounts clean_orders = orders_df.dropna(subset=["order_id", "amount"]) # drop unusable rows orders_df["discount_code"] = orders_df["discount_code"].fillna("NONE") # fill safely ``` > 🔴 **Common Mistake:** calling `orders_df.groupby("city").sum()` when the `city` column has null values silently drops those rows from the result - the revenue from orders with a missing city vanishes with no warning. Use `orders_df.groupby("city", dropna=False).sum()` to keep them visible as a `NaN` group instead. ### GroupBy, aggregation, and merging - the pandas equivalent of SQL ```python # GroupBy and aggregation - equivalent to SQL's GROUP BY + SUM revenue_by_restaurant = ( orders_df[orders_df["status"] == "delivered"] .groupby("restaurant_id")["amount"] .sum() .reset_index() .rename(columns={"amount": "total_revenue"}) ) # Merging two DataFrames - equivalent to SQL JOIN restaurants_df = pd.read_csv("restaurants.csv") enriched = orders_df.merge( restaurants_df, on="restaurant_id", how="left" # keep every order even if the restaurant record is missing ) ``` > **Note:** `how="left"` in a merge keeps every row from the left DataFrame (`orders_df`) and fills in `NaN` for any restaurant fields that do not have a match. `how="inner"` would silently drop orders whose restaurant_id has no matching restaurant row - usually not what you want in a pipeline, since it hides missing data instead of surfacing it. ### Writing output ```python revenue_by_restaurant.to_csv("output/revenue_by_restaurant.csv", index=False) revenue_by_restaurant.to_parquet("output/revenue_by_restaurant.parquet") ``` > 💡 **Practice:** load a CSV of your own making with at least one duplicate row, one badly formatted date, and one null amount. Clean it using `drop_duplicates()`, `pd.to_datetime()`, and `fillna()`, then produce a `value_counts()` summary proving the cleanup worked.
A pipeline function that looks correct and a pipeline function that is verified correct are different things. **Testing** means writing code that automatically checks your code's behavior, run before the real pipeline code ever reaches production. This is different from the data quality checks covered later in this roadmap - tests check that your code behaves correctly, data quality checks verify that your data is correct. ```python # src/pipeline.py def calculate_revenue(orders: list[dict]) -> float: """Sum the amount field for every order with status 'delivered'.""" total = 0.0 for order in orders: if order.get("status") == "delivered": total += order.get("amount", 0.0) return total ``` ```python # tests/test_pipeline.py from src.pipeline import calculate_revenue def test_normal_input(): orders = [ {"status": "delivered", "amount": 500.0}, {"status": "delivered", "amount": 750.0}, {"status": "cancelled", "amount": 200.0}, ] assert calculate_revenue(orders) == 1250.0 def test_empty_input(): # An empty list is a real scenario - an API that returns no orders for a slow day assert calculate_revenue([]) == 0.0 def test_missing_amount_field(): # Malformed source data should not crash the function orders = [{"status": "delivered"}] # no "amount" key at all assert calculate_revenue(orders) == 0.0 def test_all_cancelled(): orders = [{"status": "cancelled", "amount": 999.0}] assert calculate_revenue(orders) == 0.0 ``` ```bash ## Run every test file under tests/ - pytest finds them automatically by filename pytest tests/ -v ``` Expected output: ```text tests/test_pipeline.py::test_normal_input PASSED tests/test_pipeline.py::test_empty_input PASSED tests/test_pipeline.py::test_missing_amount_field PASSED tests/test_pipeline.py::test_all_cancelled PASSED ``` > 📌 **Remember:** the cases worth testing are not just the happy path. Empty input, missing fields, and unexpected values are exactly the situations that break pipelines in production - if you only test the normal case, you have not actually tested the function's reliability. > 💡 **Practice:** write tests for the `parse_amount()` function from earlier in this module - cover a normal numeric string, an unparseable string like `"N/A"`, and a `None` input.
It is 2 AM at Swiggy. A script that pulls yesterday's order data into the warehouse has failed silently for three nights...
Before writing a single transformation, you need a project that runs the same way on your laptop, your teammate's laptop...
Every pipeline, no matter how complex, is built from the same small set of Python primitives used correctly and consiste...
Reading and writing CSV and JSON without pandas Environment variables - never hardcode a secret in pipeline code A secre...
pandas is a Python library built around the DataFrame, a table-shaped structure with rows and columns, similar to a spre...
A pipeline function that looks correct and a pipeline function that is verified correct are different things. Testing me...
A script you run once in a notebook and a script that runs unattended every night at 2 AM need to be written differently...
requests - GET, POST, headers, authentication Pagination - looping through pages until all data is retrieved > 💡 Tip: a...
Write a Python function revenuebyrestaurant(orders: list[dict]) -> dict that reads a CSV file of Swiggy order data (colu...
Task Code Pattern Read CSV into DataFrame pd.readcsv("file.csv") Filter rows df[df["status"] == "delivered"] Group and a...
Using print() for debugging in production pipeline code leaves no timestamp, no severity level, and no way to suppress i...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.