### Why a support bot missed the obvious answer A Meesho seller support agent gets asked "my payout is stuck." The knowledge base has an article titled "Resolving delayed settlement issues." A keyword search for "payout stuck" matches nothing. Not one word overlaps. The agent replies with a generic fallback, and the seller opens a ticket instead. This is not a bug in the search code. It is a structural limit of keyword matching. Keyword search compares strings. It cannot know that "payout" and "settlement" mean nearly the same thing in this context, or that "stuck" and "delayed" describe the same problem from two different angles. **Embeddings** solve this by representing meaning as numbers instead of matching text as strings. Once meaning is a number, "payout stuck" and "delayed settlement" can be compared mathematically, even though they share zero words. ### What an embedding actually is An embedding is a list of numbers, called a vector, that represents the meaning of a piece of text. A short sentence might become a list of 768 or 1536 numbers depending on the model. Think of it like GPS coordinates, but for meaning instead of physical location. Two cities close on a map have coordinates that are numerically close. Two pieces of text with similar meaning tend to produce vectors that are close under the similarity metric used by the embedding system, even if the words themselves are completely different. > 📌 **Engineering Decision:** Use keyword search when the user is searching for an exact term, code, or ID (an order number, an error code). Use embeddings when the user is describing a problem in their own words and the right answer might use entirely different vocabulary. The model that produces these vectors learned this representation by training on huge amounts of text. It noticed that "payout" and "settlement" show up in similar contexts across millions of documents, so it placed their vectors close together. Nobody hand-coded that relationship. It emerged from the training data. ### Where embeddings fit in the bigger system you are building This module is the infrastructure layer underneath the RAG module that comes next. RAG cannot retrieve the right document unless that document was embedded correctly, chunked correctly, and indexed correctly first. Every mistake made here becomes an invisible retrieval failure a chunk later. This is why AI engineers treat this pipeline as a first-class piece of the system, not a preprocessing script you write once and forget.
### Choosing between an API model and a local model There are two practical ways to generate embeddings. Call a hosted embedding API, or run an open-weight model locally with a library like Sentence Transformers. * **Hosted API embeddings**: no infrastructure to manage, generally strong quality, costs money per token, sends your text to a third party. * **Local embeddings (Sentence Transformers)**: no per-token API charge and your data can stay inside your own infrastructure, but you still pay for compute, storage, model serving, and ongoing maintenance, it is not actually free, just a different kind of cost. > 📌 **Engineering Decision:** Hosted API embeddings for most product work, since quality and simplicity usually outweigh the per-token cost. Local embedding models when the data is sensitive and cannot leave your infrastructure, or when embedding volume is so high that API cost becomes the dominant expense. Here is generating an embedding through a hosted API: ```python from openai import OpenAI client = OpenAI() def get_embedding(text: str, model: str = "text-embedding-3-small") -> list[float]: """ Convert a string into its vector embedding. Normalizing whitespace keeps input consistent. Do not assume any single preprocessing rule universally improves quality; validate your preprocessing choices against the embedding model you actually use. """ cleaned_text = " ".join(text.split()) response = client.embeddings.create(input=[cleaned_text], model=model) return response.data[0].embedding ``` > **Note:** `text-embedding-3-small` returns a vector of 1536 numbers by default. That number is called the embedding's dimensionality. Different models produce different dimensionalities, and you cannot mix vectors from two different models in the same similarity comparison. And the equivalent using a local Sentence Transformers model: ```python from sentence_transformers import SentenceTransformer # Loads once, reused for every embedding call. Loading per-request is # a common performance mistake, since model load is far slower than inference. model = SentenceTransformer("all-MiniLM-L6-v2") def get_local_embedding(text: str) -> list[float]: """ Convert a string into its vector embedding using a locally hosted model. Returns a plain Python list so it can be stored in pgvector directly. """ embedding = model.encode(text) return embedding.tolist() ``` > 🔴 **Common Mistake:** Switching embedding models partway through a project without re-embedding everything already indexed. Vectors from two different models are not comparable, even if they happen to be the same length. Mixing them silently returns nonsense similarity scores with no error to warn you.
### Cosine similarity in plain terms Once text is a vector, you need a way to measure how close two vectors are. The standard choice is **cosine similarity**, which measures the angle between two vectors rather than the straight-line distance between them. A cosine similarity of 1 means the vectors point in exactly the same direction. In practice, higher cosine similarity generally indicates greater semantic similarity, but it does not prove two texts have identical meaning. A score near -1 means opposite meaning. A score near 0 means the vectors are close to orthogonal; whether that represents true semantic irrelevance depends on the embedding model and corpus, so treat 0 as a rough reference point, not a fixed rule. > **Note:** Cosine similarity ignores vector length and only cares about direction. This matters because two pieces of text can produce vectors of different magnitude even when they mean nearly the same thing, and cosine similarity correctly treats them as similar anyway. ```python def cosine_similarity(vector_a: list[float], vector_b: list[float]) -> float: """ Returns a score between -1 and 1. Closer to 1 means more similar meaning. """ dot_product = sum(a * b for a, b in zip(vector_a, vector_b)) magnitude_a = sum(a ** 2 for a in vector_a) ** 0.5 magnitude_b = sum(b ** 2 for b in vector_b) ** 0.5 return dot_product / (magnitude_a * magnitude_b) ``` Whether vectors need explicit normalisation before this comparison depends on the embedding model and the similarity metric your vector database uses. The real risk is not skipping normalisation everywhere by default, it is using inconsistent preprocessing between the step where you index documents and the step where you query them. If one path normalises and the other does not, similarity scores quietly become wrong.
### Why a Python list stops scaling Comparing a query vector against 50 stored vectors in a Python loop works fine. Comparing it against 5 million vectors in a loop takes seconds per query, which is unacceptable for a live product. Exact search is often perfectly fine for small datasets, the point at which it becomes a problem is measured, not assumed. A **vector database** provides storage and efficient similarity search for embeddings, answering "which stored vectors are closest to this one" quickly, even at millions of rows, using an index structure instead of comparing against every row one at a time. pgvector provides these capabilities inside PostgreSQL itself, as an extension, rather than requiring a separate standalone database system, which is exactly why it fits so naturally into a stack that already runs Postgres. > 💡 **Tip:** Think of a vector database as a specialised search structure sitting on top of your data, the same way a normal database index speeds up a `WHERE` clause without you scanning every row. ### Approximate Nearest Neighbour search Vector databases do not usually guarantee the mathematically perfect closest match. They use **Approximate Nearest Neighbour (ANN)** search, trading a small amount of accuracy for substantially lower query latency. Think of it like asking a librarian who knows the shelves well versus checking every single book in the building: the librarian might occasionally miss the one best match, but gets you a very good match almost instantly. In a production system, the right tradeoff between accuracy and speed depends on your latency budget and retrieval-quality requirements, so benchmark recall and latency on your own representative data rather than assuming a fixed accuracy target. The two most common index strategies you will encounter in pgvector are IVFFlat and HNSW. * **IVFFlat** groups vectors into clusters using k-means, like organising a library into genre sections, then searches only the closest sections. Building the index computes a set of centroids from the data present at build time, so those centroids should represent the real distribution of what you are indexing. It builds fast and uses less memory, but degrades if the data distribution shifts significantly after the index is built. * **HNSW** builds a layered graph, like a highway system with express lanes for long jumps and local roads for fine-grained search. It typically gives a strong speed/recall tradeoff and supports incremental inserts without requiring a full rebuild, though index performance and memory usage should still be monitored as the dataset keeps changing over the life of the system. The tradeoff is a slower index build and higher memory use. > 📌 **Engineering Decision:** HNSW is a strong default for many production workloads because it needs less tuning and handles data that keeps changing, but benchmark both index types against your own data and query patterns rather than assuming one is universally correct. Reach for IVFFlat specifically when the dataset is very large and mostly static, and build time or memory footprint is the dominant cost, not query accuracy.
### Why pgvector is the default for this course There are several dedicated vector databases (Pinecone, Weaviate, Qdrant, Milvus) built purely for vector search at very large scale. This module teaches **pgvector**, a Postgres extension, as the default, because most AI engineering teams are already running Postgres for the rest of their application data, and adding a second specialised database before you actually need one adds real operational cost for no benefit yet. > 📌 **Engineering Decision:** Use pgvector when you already operate Postgres and your workload fits comfortably within the database's storage, memory, latency, and throughput requirements. Move to a dedicated vector database when measured workload characteristics, not an arbitrary vector count, justify the added operational complexity. A few million vectors can be perfectly fine on pgvector depending on dimensionality, hardware, and query rate, and a much smaller dataset can be a poor fit under a very tight latency budget. Pinecone, Weaviate, and similar dedicated vector databases exist for teams that need horizontal scaling across many nodes, built-in multi-tenancy at massive scale, or advanced hybrid search out of the box. Know these exist and what they solve. This course does not teach them in depth, because teaching several vector databases shallowly is worse than teaching one properly. ### When semantic search alone is not enough Pure semantic search is not always the right tool. A query like `ERR_CONNECTION_RESET` or an exact order ID is better served by traditional keyword search, since embeddings can blur an exact code into something "close but not it." A query like "my payout hasn't arrived" is exactly where semantic search shines, since the right document may use entirely different words. **Hybrid search** combines both: a keyword/BM25-style search for exact terms and a vector search for meaning, with their results merged or re-ranked together. Most production retrieval systems that handle real, varied user queries end up needing some form of hybrid search rather than pure semantic search alone. This module gives you the vector half. The RAG module that follows covers combining it with keyword search properly. ### Setting up pgvector and storing your first embeddings ```bash ## Enable the extension inside your Postgres database psql -d prod-mumbai -c "CREATE EXTENSION IF NOT EXISTS vector;" ``` ```sql -- A table to hold Zomato-style restaurant review chunks with their embeddings CREATE TABLE review_chunks ( id SERIAL PRIMARY KEY, restaurant_id INT NOT NULL, chunk_text TEXT NOT NULL, embedding VECTOR(1536), -- must match your embedding model's dimensionality, e.g. 1536 for text-embedding-3-small, 384 for all-MiniLM-L6-v2 created_at TIMESTAMP DEFAULT NOW() ); ``` > **Note:** The dimensionality in `VECTOR(n)` is not a fixed number, it must exactly match whatever embedding model you generate vectors with. This module's hands-on lab later uses `all-MiniLM-L6-v2`, which outputs 384 dimensions, so that lab's table uses `VECTOR(384)` instead of the 1536 shown here for a hosted API model. Mismatching this number against your actual model is a common source of insert failures. ```sql -- Build an HNSW index for fast approximate nearest neighbour search CREATE INDEX ON review_chunks USING hnsw (embedding vector_cosine_ops); ``` > **Note:** `vector_cosine_ops` tells pgvector to build the index for cosine similarity specifically. pgvector also supports Euclidean distance and inner product operators. Choose a distance metric appropriate for the embedding model and validate it empirically against your retrieval workload. Cosine similarity is a common default for text embeddings, but it is not universally optimal. Whichever metric you choose, use the same one consistently at query time. ```sql -- Find the 5 review chunks closest in meaning to a customer's question SELECT chunk_text, restaurant_id, 1 - (embedding <=> '[0.021, -0.114, ...]') AS similarity FROM review_chunks ORDER BY embedding <=> '[0.021, -0.114, ...]' LIMIT 5; ``` > **Note:** The `<=>` operator is pgvector's cosine distance operator. Since distance and similarity move in opposite directions, `1 - distance` converts it back into a similarity score between 0 and 1 that is easier to reason about.
### Why "split and embed" is not a pipeline A lot of beginner tutorials show a document going through two steps: split it into chunks, embed each chunk. That is not what a production ingestion pipeline actually looks like, and skipping the missing steps is where most real-world retrieval quality problems come from. There are actually two separate pipelines here, running at different times, and it helps to keep them mentally separate. **Ingestion** happens offline, ahead of time, whenever a document is added or changed. **Retrieval** happens online, in the few hundred milliseconds after a user asks a question. The ingestion pipeline looks like this: Sources -> Parsing / OCR -> Cleaning -> PII Detection and Redaction -> Enrichment and Metadata -> Chunking -> Embedding -> Indexing The retrieval pipeline, which runs later at query time, looks like this: User Query -> Query Embedding -> Search (vector or hybrid) -> Metadata Filtering -> Ranking -> Evaluation Each ingestion stage exists because of a specific failure mode that shows up without it. Skipping cleaning means garbage HTML tags get embedded alongside real content. Skipping PII redaction means sensitive data becomes permanently searchable, covered in detail below. Skipping metadata means you can retrieve a chunk but never know which document, version, or date it came from. On the retrieval side, skipping evaluation means you find out retrieval quality is bad only when a user complains, not before. ### Document loaders and OCR awareness Real company knowledge lives in mixed formats: PDFs, Word documents, Notion pages, Slack exports, scanned images of physical documents. A **document loader** is the code responsible for extracting clean text out of each of these formats. Scanned documents and image-based PDFs contain no extractable text at all, only pixels. These require **OCR (Optical Character Recognition)** to convert the image into text before anything downstream can work. Know that this step exists and when you need it. A document that returns an empty string from your parser is often a scanned PDF silently failing, not an empty document. ### Chunking strategy You cannot embed an entire 40-page policy document as a single vector and expect useful retrieval. A single embedding tends to represent a document at a relatively coarse level, which makes retrieval of a specific detail buried on page 22 unreliable. > 🔴 **Common Mistake:** Chunking by a fixed character count with no regard for natural boundaries. Splitting exactly every 500 characters can cut a sentence in half, separating a policy's condition from its exception, and both halves become individually confusing when retrieved alone. Better chunking respects natural document structure, splitting on paragraph or section boundaries where possible, and often overlaps chunks slightly (for example, the last two sentences of one chunk repeated as the first two of the next) so that context near a boundary is not lost. ### Metadata, versioning, and incremental ingestion Every chunk stored should carry metadata alongside its vector: source document, section, last-updated date, document version, access permissions. Without this, retrieval can find the right words but you have no way to filter, cite the source, or know the information is stale. In practice, a `metadata JSONB` column is a convenient way to store this alongside dedicated columns for the fields you filter on most often, such as `source_document`, `document_version`, and `updated_at`. A production retrieval query typically filters on this metadata at the same time as it searches by similarity, for example matching semantic closeness while also requiring `tenant_id = current_tenant` and `access_level <= user_access_level`. Retrieval is not just a relevance mechanism, it is also an authorization boundary, and metadata filtering is how that boundary gets enforced. Production systems should also record the embedding model and embedding version used for each indexed vector, for example `embedding_model: "all-MiniLM-L6-v2"`. If you later switch models, this lets you identify exactly which vectors came from the old model and need re-embedding, instead of silently treating old and new vectors as interchangeable when they are not comparable at all. **Incremental ingestion** means re-processing only documents that actually changed, instead of re-embedding your entire knowledge base every time one policy document is edited. A naive full re-embed on every change is slow and expensive at scale, and becomes the reason teams stop updating their knowledge base regularly. ```python def needs_reembedding(document_path: str, stored_hash: str) -> bool: """ Compares a fresh hash of the document's content against the hash stored from the last successful ingestion run. Only documents that actually changed get re-embedded. In a full pipeline, a changed document also means deleting or replacing that document's old chunks before inserting the newly parsed and re-embedded ones, not just adding new chunks alongside stale ones. """ import hashlib with open(document_path, "rb") as f: current_hash = hashlib.sha256(f.read()).hexdigest() return current_hash != stored_hash ``` ### Deduplication Real document sets contain near-duplicates: the same policy PDF uploaded twice with a different filename, or a Slack thread quoting an entire earlier message. Embedding both wastes storage and, worse, can cause retrieval to return the same content twice while missing a genuinely different relevant chunk that got pushed out of the top results. > 💡 **Tip:** Near-duplicate detection can use embeddings as one signal, treating a very high cosine similarity between two chunks as a candidate for duplication, not proof of it. Values like 0.97 to 0.99 are a reasonable starting point for experimentation, not a universal production threshold. For production systems, combine semantic similarity with a normalized text hash or shared document ID where available, since high similarity alone does not guarantee two chunks are actually duplicates. ### PII detection and redaction before embedding If a document contains a customer's phone number, email, or bank account details, that information gets embedded and stored the moment you run it through this pipeline, and it can resurface later if retrieval pulls that chunk into a response. > ⚠️ **Security:** Run PII detection and redaction before embedding, not after. Once sensitive data is embedded and indexed, deleting the source document does not remove the fact that a vector representing that sensitive content still exists and is still searchable. Redaction is not a substitute for authorization. Even a perfectly redacted corpus can leak sensitive information across users if retrieval has no concept of who is allowed to see what. If User A's query can retrieve a chunk sourced from User B's private document, redacting PII inside that chunk does not fix the underlying leak. Document- and tenant-level access controls, enforced through the metadata filtering covered above, are what actually prevent that class of failure. Treat redaction and access control as two separate, both-mandatory layers, not one solving the other. ```python import re def redact_pii(text: str) -> str: """ Basic pattern-based redaction for common PII types before embedding. Production systems typically pair this with a dedicated PII detection model for names and addresses, which regex alone cannot reliably catch. """ text = re.sub(r"\b\d{10}\b", "[REDACTED_PHONE]", text) text = re.sub(r"[\w.+-]+@[\w-]+\.[a-zA-Z]{2,}", "[REDACTED_EMAIL]", text) return text ``` > **Note:** Regex catches predictable patterns like a 10-digit phone number or an email address format. It does not reliably catch names, addresses, or context-dependent sensitive information. Production pipelines usually combine regex for structured patterns with a dedicated PII detection model for everything else.
Why a support bot missed the obvious answer A Meesho seller support agent gets asked "my payout is stuck." The knowledge...
Choosing between an API model and a local model There are two practical ways to generate embeddings. Call a hosted embed...
Cosine similarity in plain terms Once text is a vector, you need a way to measure how close two vectors are. The standar...
Why a Python list stops scaling Comparing a query vector against 50 stored vectors in a Python loop works fine. Comparin...
Why pgvector is the default for this course There are several dedicated vector databases (Pinecone, Weaviate, Qdrant, Mi...
Why "split and embed" is not a pipeline A lot of beginner tutorials show a document going through two steps: split it in...
Concept checks > 💡 Practice: Before running any code, answer these for yourself. Why does cosine similarity ignore vect...
Decision Recommendation Embedding model source Hosted API by default, local model for sensitive or high-volume data Simi...
Embedding an entire long document as a single vector happens when a team wants to move fast and skips chunking entirely....
Install dependencies and enable pgvector. Create the table and index. > Note: This table implements the metadata practic...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.