Your fraud detection model hit 98% accuracy in testing. Your manager was thrilled. Three weeks after launch, it is missing obvious fraud and flagging normal transactions from regular Zomato customers in Bengaluru. Nobody touched the model. The bug was never in the model. It was in a `groupby()` that averaged a column across the full dataset before the train/test split, quietly leaking future information into the training rows. It was in a `SELECT *` that pulled 40 columns nobody checked for nulls. It was in a `merge()` that silently duplicated rows because a customer ID was not unique. AI engineers do not get to blame the model when the data was never clean or correctly split in the first place. This module is about building that discipline before you ever touch an LLM. ### Why this comes before LLMs and RAG Every retrieval pipeline you will build later ingests from somewhere. Every evaluation dataset you will build later is a table. Every fine-tuning dataset is rows and columns before it is JSON. If you cannot clean, join, and split data correctly here, every later module inherits the mistake silently.
### Why a Python list is not enough for numerical work A Python list stores each number as a full Python object, scattered in memory with pointers connecting them. A NumPy array stores raw numbers side by side in one contiguous block, the way a spreadsheet stores a column. That difference is why a loop over a list of a million numbers is slow, and the same operation on a NumPy array is not. NumPy pushes the loop down into optimized C code instead of running it one Python object at a time. ```python import numpy as np ## A plain Python list requires a manual loop to double every value prices = [499, 899, 1299, 2499] doubled_list = [p * 2 for p in prices] ## A NumPy array applies the operation to every element at once prices_arr = np.array(prices) doubled_arr = prices_arr * 2 # vectorized - no explicit loop print(doubled_arr) ``` ```text [ 998 1798 2598 4998] ``` > **Note:** "Vectorized" means the operation runs across the whole array in one call > instead of you writing a `for` loop. This matters because vectorized NumPy code is > typically 10 to 100 times faster than the equivalent Python loop on large arrays, > and you will lean on this constantly once you are batching embeddings or scores. ### Creating and reshaping arrays `np.array()` builds an array from a Python list or nested list. `reshape()` changes how the same values are arranged into rows and columns without copying the data. Creation Reshape np.array([1,2,3,4,5,6]) .reshape(2, 3) +---+---+---+---+---+---+ +---+---+---+ | 1 | 2 | 3 | 4 | 5 | 6 | --> | 1 | 2 | 3 | +---+---+---+---+---+---+ +---+---+---+ | 4 | 5 | 6 | +---+---+---+ ```python import numpy as np ## A flat list of six order amounts orders = np.array([1, 2, 3, 4, 5, 6]) ## Reshape into 2 rows x 3 columns - useful when grouping a flat batch ## of embeddings or scores into a matrix shape orders_matrix = orders.reshape(2, 3) print(orders_matrix) ``` ```text [[1 2 3] [4 5 6]] ``` ### Generating random arrays for testing pipelines `np.random.rand()` generates uniform random floats between 0 and 1. `np.random.randint()` generates random integers in a given range. Both are useful for generating dummy data to test a pipeline before real data is available. ```python import numpy as np ## 2x3 array of random floats between 0 and 1 - good for fake similarity scores scores = np.random.rand(2, 3) ## 2x2 array of random integers between 0 and 10 - good for fake order counts counts = np.random.randint(0, 10, size=(2, 2)) print(scores) print(counts) ``` ```text [[0.63 0.91 0.40] [0.27 0.76 0.10]] [[3 7] [1 9]] ```
### Element-wise math versus matrix multiplication Adding two arrays with `+` combines them position by position. This is different from `np.dot()`, which performs true matrix multiplication, the operation that powers every neural network layer you will meet in later modules. ```python import numpy as np a = np.array([[1, 2], [3, 4]]) b = np.array([[5, 6], [7, 8]]) ## Element-wise addition - each position is added independently elementwise_sum = a + b print(elementwise_sum) ``` ```text [[ 6 8] [10 12]] ``` ```python ## Matrix multiplication - rows of a combined with columns of b matrix_product = np.dot(a, a) print(matrix_product) ``` ```text [[ 7 10] [15 22]] ``` > **Note:** You will not hand-write matrix multiplication in real AI engineering work, > a deep learning framework does that for you. What matters here is recognizing the > operation when you see it in documentation, and knowing `np.dot()` is not the same > as `+`. ### Aggregation functions `sum()`, `mean()`, `max()`, and `min()` collapse an array down to a single summary number, or a row/column of summaries if you specify an axis. ```python import numpy as np order_values = np.array([499, 899, 1299, 2499, 199]) print("Mean order value:", order_values.mean()) print("Highest order:", order_values.max()) print("Total revenue:", order_values.sum()) ``` ```text Mean order value: 1079.0 Highest order: 2499 Total revenue: 5395 ``` ### A working-level look at linear algebra functions You do not need to derive these by hand. You need to recognize them and know what each is for. | Function | What it does | |:---|:---| | `np.dot(a, b)` | Matrix or vector multiplication | | `np.linalg.inv(a)` | Computes the inverse of a matrix | | `np.inner(a, b)` | Inner product of two arrays | | `np.outer(a, b)` | Outer product, builds a matrix from two vectors | | `np.linalg.eig(a)` | Eigenvalues and eigenvectors of a matrix | > 📌 **Remember:** Matrix multiplication (`np.dot`) is the operation underneath every > layer of a neural network, and it is also what happens when you compute similarity > between two embedding vectors later in this roadmap. You will keep meeting it.
### Why pandas instead of raw NumPy for real datasets NumPy arrays are fast but hold only one data type, and have no concept of column names. Real data has mixed types (text, numbers, dates) and needs labels. **pandas** gives you the `DataFrame`, a labeled table on top of NumPy, closer to a spreadsheet than a raw array. ```python import pandas as pd ## Reading a Zomato-style order export orders_df = pd.read_csv("zomato_orders_mumbai.csv") print(orders_df.head()) ``` ```text order_id restaurant city order_value rating 0 1001 Punjab Grill Mumbai 650.0 4.5 1 1002 Sagar Ratna Mumbai 420.0 4.2 2 1003 Punjab Grill Pune 310.0 NaN 3 1004 Truffles Mumbai 890.0 4.7 4 1005 Sagar Ratna Pune 275.0 3.9 ``` ### Inspecting before trusting Before doing anything with a new dataframe, check its shape, types, and missing values. Skipping this step is how nulls quietly break a downstream pipeline three steps later. ```python import pandas as pd orders_df = pd.read_csv("zomato_orders_mumbai.csv") ## Rows and columns as a (rows, cols) tuple print(orders_df.shape) ## Data types and non-null counts per column, in one call orders_df.info() ## Count of missing values per column print(orders_df.isnull().sum()) ``` ```text (5000, 5) <class 'pandas.core.frame.DataFrame'> RangeIndex: 5000 entries, 0 to 4999 # Column Non-Null Count Dtype --- ------ -------------- ----- 0 order_id 5000 non-null int64 1 restaurant 5000 non-null object 2 city 5000 non-null object 3 order_value 4820 non-null float64 4 rating 4390 non-null float64 order_value 180 rating 610 dtype: int64 ``` ### Cleaning missing and duplicate values Two real strategies exist for missing data: drop it, or fill it with a reasonable value like the column mean. Which one is correct depends on how much data you would lose and whether the missing value is random or meaningful. ```python import pandas as pd orders_df = pd.read_csv("zomato_orders_mumbai.csv") ## Strategy 1: drop rows missing order_value - fine when the loss is small clean_df = orders_df.dropna(subset=["order_value"]) ## Strategy 2: fill missing rating with the column mean instead of dropping mean_rating = orders_df["rating"].mean() orders_df["rating"] = orders_df["rating"].fillna(mean_rating) ## Remove exact duplicate rows, common after a join or a re-run export orders_df = orders_df.drop_duplicates() print(orders_df.isnull().sum()) ``` ```text order_id 0 restaurant 0 city 0 order_value 0 rating 0 dtype: int64 ``` > 🔴 **Common Mistake:** Filling every missing numeric column with its mean by > default, without asking why the value is missing. A missing `rating` because the > customer skipped rating is not the same as a missing `order_value` because of a > broken export job. The second case may mean the row should be dropped, not filled. ### Grouping and merging `groupby()` aggregates rows that share a value in one or more columns. `merge()` combines two dataframes based on a shared key column, the pandas equivalent of a SQL JOIN. ```python import pandas as pd orders_df = pd.read_csv("zomato_orders_mumbai.csv") ## Average order value per city avg_by_city = orders_df.groupby("city")["order_value"].mean() print(avg_by_city) ``` ```text city Mumbai 687.4 Pune 512.9 Name: order_value, dtype: float64 ``` ```python customers_df = pd.read_csv("zomato_customers.csv") ## Merge orders with customer details on a shared customer_id column ## how="left" keeps every order even if a matching customer is missing merged_df = orders_df.merge(customers_df, on="customer_id", how="left") print(merged_df.shape) ``` > 🔴 **Common Mistake:** Merging on a column that is not actually unique in one of > the two tables. If `customer_id` appears twice in `customers_df` for the same > customer, every matching order row in `orders_df` gets silently duplicated in the > result. Always check `customers_df["customer_id"].is_unique` before merging on it.
### Why plotting comes before modeling, not after A summary statistic can hide a broken dataset. A column can have a correct-looking mean while actually containing two separate clusters, or a handful of extreme outliers dragging the average. Plotting the data catches this in seconds, reading `.describe()` output alone often does not. ```python import matplotlib.pyplot as plt ## Histogram reveals the shape of order_value - is it one cluster or several? orders_df["order_value"].plot.hist(bins=30) plt.xlabel("Order Value (INR)") plt.title("Distribution of Order Values - Mumbai") plt.show() ``` > **Note:** `plt.show()` is required outside Jupyter to actually render the window. > Inside a Jupyter notebook, add `%matplotlib inline` once at the top of the notebook > instead, so plots render inline automatically without calling `.show()` each time. ### Boxplots for spotting outliers fast A boxplot shows the median, the middle 50% of values, and flags points far outside that range as individual dots, outliers you would otherwise miss in a raw average. ```python import matplotlib.pyplot as plt ## Boxplot of order_value split by city - outliers appear as isolated dots orders_df.boxplot(column="order_value", by="city") plt.title("Order Value Spread by City") plt.suptitle("") plt.show() ``` > 💡 **Tip:** If a boxplot shows a handful of orders worth 50,000 rupees sitting far > above everything else in a dataset of mostly 300 to 900 rupee orders, do not just > delete them. Confirm first whether they are real bulk catering orders or a data > entry bug, deleting real data because it looks unusual is its own mistake.
### Why SQL matters even if you mostly write Python Training and evaluation data rarely starts as a clean CSV on your laptop. It lives in a production database with millions of rows. Pulling only the rows and columns you need with SQL, before loading anything into pandas, is faster and avoids crashing your machine trying to load an entire table into memory. ```sql -- Basic filtering: recent high-value orders only SELECT order_id, customer_id, order_value, order_date FROM orders WHERE order_value > 1000 AND order_date >= '2026-01-01'; ``` > 🔴 **Common Mistake:** Running `SELECT *` on a production table out of habit. On a > table with forty columns and ten million rows, this pulls far more data than the > task needs, slows the query, and often exceeds available memory once it reaches > pandas. Select only the columns you will actually use. ### JOINs for combining related tables A JOIN combines rows from two tables based on a matching column, most commonly an ID shared between them. `INNER JOIN` keeps only rows that match in both tables. `LEFT JOIN` keeps every row from the left table even without a match. ```sql -- Top customers by total spend, joining orders to customer details SELECT c.customer_id, c.customer_name, c.city, SUM(o.order_value) AS total_spend FROM customers AS c INNER JOIN orders AS o ON c.customer_id = o.customer_id WHERE c.city = 'Bengaluru' GROUP BY c.customer_id, c.customer_name, c.city ORDER BY total_spend DESC LIMIT 10; ``` > **Note:** `INNER JOIN` silently drops any customer with zero orders, since there > is nothing on the `orders` side to match. If the goal is "every customer, with > their spend or zero if they have never ordered," `LEFT JOIN` is the correct choice > instead, `INNER JOIN` would quietly remove them from the result. | JOIN type | Keeps | |:---|:---| | `INNER JOIN` | Only rows matching in both tables | | `LEFT JOIN` | All rows from the left table, matched or not | | `RIGHT JOIN` | All rows from the right table, matched or not | ### Validating data quality directly in SQL Checking for nulls, duplicates, and out-of-range values in SQL before pulling data into Python catches problems at the source, closer to where they were introduced. ```sql -- Null check on a column that should never be empty SELECT COUNT(*) AS missing_order_value FROM orders WHERE order_value IS NULL; -- Duplicate check on what should be a unique order_id SELECT order_id, COUNT(*) AS occurrences FROM orders GROUP BY order_id HAVING COUNT(*) > 1; -- Range check for an impossible negative order value SELECT order_id, order_value FROM orders WHERE order_value < 0; ```
Your fraud detection model hit 98% accuracy in testing. Your manager was thrilled. Three weeks after launch, it is missi...
Why a Python list is not enough for numerical work A Python list stores each number as a full Python object, scattered i...
Element-wise math versus matrix multiplication Adding two arrays with + combines them position by position. This is diff...
Why pandas instead of raw NumPy for real datasets NumPy arrays are fast but hold only one data type, and have no concept...
Why plotting comes before modeling, not after A summary statistic can hide a broken dataset. A column can have a correct...
Why SQL matters even if you mostly write Python Training and evaluation data rarely starts as a clean CSV on your laptop...
What a train/test split actually protects against A model is only useful if it performs well on data it has never seen. ...
Work through these four steps in order. Each step depends on the one before it. Clean a Zomato-style order dataset. Load...
Task Tool / Command Load a CSV into a dataframe pd.readcsv("file.csv") Check nulls per column df.isnull().sum() Drop row...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.