Math and Statistics for AI Engineers

### Vectors: data as a list of numbers A **vector** is just an ordered list of numbers. That's it. `[0.2, -1.4, 0.9]` is a vector with 3 dimensions. In AI, everything gets turned into vectors before a model can touch it - a word, a sentence, an image, a user's purchase history. This is not optional; matrix multiplication (the operation every neural network layer runs) only works on numbers, not raw text or pixels. > 💡 **Tip:** Think of a vector like GPS coordinates for meaning. Just as two physical locations close on a map are physically near each other, two pieces of text with similar meaning produce vectors that sit close together mathematically. This is the entire idea behind embeddings, which you'll use constantly later in this roadmap. ### Matrices: a table of vectors A **matrix** is a 2D grid of numbers - a stack of vectors. A batch of 32 sentences, each turned into a 768-number vector, is a 32x768 matrix. A dataset of 10,000 customer rows with 15 features each is a 10,000x15 matrix. ```python import numpy as np ## A batch of 3 embeddings, each with 4 dimensions ## In a real LLM these would be 768, 1536, or larger embeddings = np.array([ [0.12, -0.45, 0.88, 0.03], ## "refund" [0.15, -0.41, 0.85, 0.02], ## "return" - close to "refund" [-0.9, 0.31, -0.12, 0.77], ## "pizza" - far from both ]) print(embeddings.shape) ## (3, 4) -> 3 rows, 4 columns ``` > **Note:** `.shape` tells you the matrix dimensions as `(rows, columns)`. This is the single most useful debugging line in any AI codebase - a huge fraction of real bugs are shape mismatches between what one layer outputs and what the next layer expects. ### Matrix multiplication: why every layer needs it Matrix multiplication is one of the fundamental operations behind neural networks. Linear layers, attention projections, and many other parts of modern models rely heavily on it - alongside other operations like normalization, activations, and softmax. ```python ## A tiny "layer": 4 input features -> 2 output features weights = np.array([ [0.5, -0.2], [0.1, 0.4], [0.3, 0.6], [-0.1, 0.2], ]) ## One input vector with 4 features input_vector = np.array([1.0, 0.5, -0.3, 0.8]) ## Matrix multiplication - the core operation of a forward pass output = input_vector @ weights print(output) ## a new vector with 2 features ``` > 📌 **Engineering Decision:** You will almost never write matrix multiplication by hand in real work - PyTorch, NumPy, and every ML framework handle it. What you need is the mental model: a layer is "input vector times weight matrix," so when a model's output shape doesn't match what you expected, the weight matrix dimensions are usually the first place to check. ### Dot product and cosine similarity A **dot product** combines two vectors into a single number - one of the basic ways AI systems measure how strongly two vectors point in the same direction. ```python import numpy as np a = np.array([1, 2, 3]) b = np.array([2, 1, 4]) print(np.dot(a, b)) ``` For embeddings, the more common similarity measure is **cosine similarity**. It measures the angle between two vectors rather than their raw size, so it stays meaningful even when vectors have very different magnitudes. * Close to 1 -> vectors point in similar directions (similar meaning) * Around 0 -> little directional similarity * Close to -1 -> opposite directions This is why cosine similarity appears constantly in semantic search and RAG systems, which you'll build later in this roadmap. > 📌 **Remember:** You don't need to calculate cosine similarity by hand. You need to recognize what it measures and recognize it when you see it in an embedding or retrieval system. ### Tensors: the data structure AI frameworks actually use A **tensor** is a generalization of vectors and matrices to more dimensions. * Scalar -> 0 dimensions (a single number) * Vector -> 1 dimension * Matrix -> 2 dimensions * Tensor -> 3 or more dimensions An image is commonly represented as `(height, width, channels)`. A batch of images becomes `(batch, height, width, channels)`. In PyTorch, tensors are the basic data structure used to store model inputs, weights, and outputs - `tensor.shape` is something you'll check constantly. > 📌 **Remember:** You don't need tensor algebra here. For AI engineering, understanding tensor shapes and dimensions matters far more than the underlying math. ### Eigenvalues and eigenvectors, briefly An **eigenvector** of a matrix is a direction that doesn't change when the matrix transforms it - it only gets stretched or shrunk. The amount of stretching is the **eigenvalue**. You will not compute these by hand in this job. The one place this awareness matters: dimensionality reduction techniques like PCA use eigenvectors to find the directions in your data that carry the most information, which is how a huge feature set gets compressed down without losing the signal that matters. > 💡 **Tip:** If you ever see a research paper mention eigenvalues in the context of model behavior or data compression, the takeaway you need is "this is finding the most important directions in the data" - not the underlying linear algebra proof. ---