### What the agent actually knows When you use a large language model like GPT or Claude, it knows a lot of things — general programming, Linux commands, networking concepts, and much more. But here is the catch: everything it knows comes from its training data, which has a cutoff date. After that date, it knows nothing new. More importantly, it has absolutely no idea about anything specific to your company or your infrastructure. Your agent does not know: * What your Kubernetes cluster looks like * How your team handles a database failover * What the runbook says for a high CPU alert on your payment service * What went wrong in last month's production incident * Which services talk to which other services in your architecture If you ask your agent "what should I do when the payment service throws a 502?" it will give you a generic answer based on general knowledge. It might even make something up that sounds correct but is completely wrong for your setup. This is called a **hallucination** — the model confidently generates a wrong answer because it has no actual information to work from. ### What this costs you during an incident Imagine it is 2 AM. An alert fires. Your on-call engineer asks the AI assistant for help. The assistant gives advice based on general best practices — but your system works differently. The engineer follows the wrong steps. Ten minutes wasted. The incident gets worse. This is not a hypothetical. It happens because the AI had no access to your actual runbooks or your system's context. ### The fix in plain English What if the agent could search your documents first, find the relevant section, and then answer using that real information instead of guessing? That is exactly what RAG does. ---
### Breaking down the name RAG stands for **Retrieval-Augmented Generation**. Three words, three jobs: * **Retrieval** — find the right documents from your knowledge base * **Augmented** — add those documents to the question before asking the AI * **Generation** — the AI generates an answer using both your question and the retrieved documents So instead of the AI answering purely from memory, it first searches your knowledge base, picks up the most relevant pieces, and uses those as context to write the answer. The answer is now grounded in your actual documentation. ### A simple example Without RAG: ``` You ask: "What do I do when the auth service returns 503?" AI answers: "A 503 means the service is unavailable. You should check if the service is running, look at recent deployments, and check resource limits..." (generic advice, might be wrong) ``` With RAG: ``` You ask: "What do I do when the auth service returns 503?" System first searches your runbooks → finds your auth-service-503.md AI answers: "According to your runbook, a 503 on the auth service usually means the Redis connection pool is exhausted. First run `kubectl get pods -n auth` to check pod health, then check Redis memory with the dashboard at monitor.internal/redis..." (your actual runbook) ``` Same question. Completely different answer quality. ### RAG vs fine-tuning — why not just train the model on your docs? This is a common question. Why not just teach the model your runbooks directly by fine-tuning it? Fine-tuning means retraining the model with new data so it learns that information permanently. The problem with using fine-tuning for ops documentation is: * Your runbooks change constantly — every time they do, you have to retrain * Retraining is expensive, slow, and requires ML expertise * Fine-tuned knowledge can get mixed up with general knowledge and cause strange outputs * You cannot easily remove something the model has learned RAG treats your knowledge base as a separate, searchable database. Update a runbook? The change is immediately available at next query — no retraining. Add a new service? Index its documentation. Remove an old one? Delete it from the index. Clean, simple, fast. ### RAG vs putting everything into the prompt Another approach people try: just copy all your documentation into the prompt every time. This breaks for a few reasons: * LLMs have a context window limit — there is only so much text they can process at once * More text in the prompt means more tokens, which means more cost per query * When you stuff too much into the prompt, the model's attention gets diluted and quality drops * Not all your documentation is relevant to every question — you only need the right pieces RAG solves this by only retrieving what is relevant to the current question. Instead of giving the model your entire 500-page ops wiki, you give it the three most relevant paragraphs. ### The flow at a glance ``` Your Question ↓ [ Find relevant documents from your knowledge base ] ↓ [ Add those documents to your question as context ] ↓ [ LLM generates answer using question + context ] ↓ Grounded Answer (based on your actual docs) ``` ---
### The fundamental problem Computers work with numbers. Text is not numbers. So before a computer can search for meaning in text — not just keywords, but actual meaning — it needs a way to convert text into numbers. Not just any numbers though. The numbers need to carry the meaning of the text, so that two pieces of text with similar meaning produce similar numbers. This is what an **embedding** is: a list of numbers that represents the meaning of a piece of text. ### What an embedding looks like When you send a sentence to an embedding model, you get back a long list of decimal numbers. For example: ``` Text: "the server is not responding" Embedding: [0.023, -0.841, 0.192, 0.007, -0.334, 0.651, ...] ... 1536 numbers total ... ``` That list of 1536 numbers is the mathematical representation of your sentence's meaning. ### Why similar text produces similar numbers Here is the key insight: the embedding model was trained on massive amounts of text. During training, it learned that certain words and concepts appear together in similar contexts. That learning gets encoded into how it maps text to numbers. So when you embed "server is not responding" and "host is unreachable", the resulting number lists will be very close to each other because those two phrases mean similar things in similar contexts. And when you embed "the recipe calls for two cups of flour", that number list will be very far from both of those, because it has nothing to do with servers. ### Measuring closeness with a number The closeness between two embeddings is measured using **cosine similarity** — a single number between -1 and 1 that tells you how similar two vectors are. You do not need to know the formula. Just understand the scale: | Score | Meaning | |-------|---------| | 1.0 | Identical meaning | | 0.9+ | Very similar | | 0.7–0.9 | Related | | 0.5–0.7 | Loosely related | | Below 0.5 | Probably unrelated | So when someone asks "why is my service slow?", the system can compute the embedding of that question, then find all document chunks whose embeddings score above 0.8 cosine similarity. Those are your most relevant results. ### Why keyword search is not enough for ops Traditional keyword search looks for exact word matches. It is fast but brittle. Consider these two sentences: * "restart the application container" * "bounce the app pod" A keyword search would say these have nothing in common — no shared words. But any DevOps engineer knows they mean the same thing. Vector search understands they mean the same thing because their embeddings are close together. This is why embeddings are essential for ops knowledge retrieval — your runbooks and your queries will often use different words to describe the same thing. ---
### The machine that creates embeddings An embedding model is a neural network trained specifically to convert text into vectors. You send it text, it sends back a vector. That is its entire job. ```python # Example: using OpenAI's embedding model from openai import OpenAI client = OpenAI() response = client.embeddings.create( model="text-embedding-3-small", input="restart the Nginx service after config change" ) vector = response.data[0].embedding print(len(vector)) # 1536 print(vector[:5]) # [0.023, -0.841, 0.192, 0.007, -0.334] ``` ### The important rule: same model for everything When you index your documents, you embed them with a specific model. When a user asks a question, you embed the question with that same model. If you use different models, the numbers live in different spaces and cannot be compared. It is like measuring one thing in metres and another in feet and trying to say which is bigger — the comparison breaks. Always use the same embedding model for both indexing and querying. ### OpenAI embeddings vs free local models You have two main choices: **OpenAI `text-embedding-3-small`** * Hosted API, no setup required * Costs money per token (very cheap — roughly ₹0.001 per 1000 tokens) * 1536 dimensions, very good quality * Best choice for production **Sentence Transformers (local, free)** * Runs on your own machine, no API calls, no cost * Model: `all-MiniLM-L6-v2` — fast, 384 dimensions, good quality * Best choice for: learning, prototyping, air-gapped environments, cost-sensitive setups ```python # Free local embedding model from sentence_transformers import SentenceTransformer model = SentenceTransformer("all-MiniLM-L6-v2") vector = model.encode("high CPU alert on payment service") print(len(vector)) # 384 ``` For this module's project, we will use the free local model so you can run everything without an API key. ---
### Why a regular database cannot do this In a normal SQL database, you search by exact match or range: ```sql SELECT * FROM alerts WHERE service = 'payment'; SELECT * FROM logs WHERE timestamp > '2024-01-01'; ``` These work great for structured data. But you cannot write a SQL query that says "find me the rows whose meaning is closest to this question." SQL does not understand meaning. A **vector database** is a database built specifically to store vectors and answer the question: "which stored vectors are most similar to this query vector?" It does this efficiently — returning results in milliseconds even when you have millions of vectors stored. ### How it finds similar vectors fast The naive approach would be: compare the query vector against every stored vector, calculate the similarity score, return the top results. This works fine for a thousand vectors. For a million vectors, it is too slow. Vector databases use an algorithm called **HNSW (Hierarchical Navigable Small World)** to make this fast. You do not need to understand how it works in detail — just know that it builds a smart graph structure over your vectors so searches skip most of the comparisons. Think of it like this: instead of checking every house on every street to find your friend's house, you use a map that says "start in this neighbourhood, then this block, then this house." Much faster. This is called **Approximate Nearest Neighbor (ANN)** search — it finds vectors that are very close to the best match, trading a tiny amount of accuracy for a huge gain in speed. ### What metadata filtering adds Every chunk you store in a vector database can have **metadata** attached — extra fields that describe the chunk. ```python # Storing a chunk with metadata collection.add( documents=["Restart the auth service pods in the auth namespace"], metadatas=[{ "source": "auth-service-runbook.md", "service": "auth", "type": "runbook", "severity": "high", "last_updated": "2024-11-01" }], ids=["chunk-001"] ) ``` You can then filter searches by metadata: ```python # Only search runbooks for the auth service results = collection.query( query_texts=["service keeps crashing"], where={"service": "auth", "type": "runbook"}, n_results=5 ) ``` This is extremely useful in ops. When an auth service alert fires, you only want to retrieve auth service documentation — not payment service runbooks. ### Vector database vs vector index You might come across the term **FAISS** (from Facebook) — this is a **vector index**, not a database. | | Vector Index (FAISS) | Vector Database (ChromaDB, Qdrant) | |---|---|---| | Stores vectors | Yes | Yes | | Stores original text | No — you manage this | Yes — built in | | Metadata filtering | No | Yes | | Persistent storage | You manage this | Built in | | Production ready | Needs a lot of extra work | Yes | For ops use cases, use a vector database. It handles everything in one place. ---
Now that you understand each piece, here is how they all connect into one complete system. ### The two phases RAG has two phases that happen at different times: **Phase 1: Indexing (done once, or whenever docs change)** This is where you process your documents and store them. **Phase 2: Querying (done every time a user asks a question)** This is where you answer questions using the indexed documents. ### Phase 1: Indexing your ops knowledge This happens once — or whenever your documents change. ``` ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ INDEXING PHASE (runs offline, once or on update) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Runbooks / Postmortems / Playbooks / Arch Docs ↓ [ 1. Chunking ] Split each doc into small pieces (300-500 words, with overlap) ↓ [ 2. Embedding ] Convert each chunk to a vector using the embedding model ↓ [ 3. Store in Vector DB ] Save vector + original text + metadata (source file, service name, doc type) ↓ Knowledge Base is ready to search ``` After this phase runs, your entire ops knowledge base is stored and searchable in milliseconds. ### Phase 2: Answering a question This happens every time a user asks something. ``` ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ QUERYING PHASE (runs live, every user question) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ User: "payment service 502, what do I check?" ↓ [ 4. Query Rewriting ] (optional) Rewrite vague question into clearer search-friendly version ↓ [ 5. Embed the Question ] Same embedding model as indexing ↓ [ 6. Vector Search ] Find top-K chunks closest in meaning ↓ [ 7. Reranking ] (optional) Reorder results by a more precise model ↓ [ 8. Build the Prompt ] Question + retrieved chunks as context ↓ [ 9. LLM ] Generates answer using your real docs ↓ Answer + Citations (source filenames) ``` ### What this looks like for an ops team Suppose you have these documents indexed: * `payment-service-runbook.md` * `database-failover-procedure.md` * `incident-2024-11-black-friday.md` * `kubernetes-troubleshooting.md` * `alert-definitions.md` When someone asks "why is payment checkout slow during high traffic?", the system retrieves the most relevant sections from those documents and gives an answer that references your actual procedures — not generic internet advice. ---
What the agent actually knows When you use a large language model like GPT or Claude, it knows a lot of things — general...
Breaking down the name RAG stands for Retrieval-Augmented Generation. Three words, three jobs: Retrieval — find the righ...
The fundamental problem Computers work with numbers. Text is not numbers. So before a computer can search for meaning in...
The machine that creates embeddings An embedding model is a neural network trained specifically to convert text into vec...
Why a regular database cannot do this In a normal SQL database, you search by exact match or range: These work great for...
Now that you understand each piece, here is how they all connect into one complete system. The two phases RAG has two ph...
What chunking is Your runbooks and docs are too long to store as single entries. If you stored an entire 10-page runbook...
Four options worth knowing pgvector — vector search inside PostgreSQL If your team already runs PostgreSQL, this is the ...
Where pure vector search breaks down Vector search is great at finding meaning. But it is not great at finding exact mat...
The mismatch problem Users ask questions the way they think. Runbooks are written the way engineers write documentation....
Knowing this saves you before it bites you Building a RAG pipeline that works in demos is straightforward. Building one ...
What you are building A command-line tool that lets you ask questions in plain English and get answers from your own run...
Why citations matter in ops When your RAG assistant gives an answer, an engineer needs to trust it. Especially during an...
A common beginner mistake RAG is powerful and it is tempting to use it for everything once you understand it. But vector...
RAG alone is not an agent Everything we have built so far is a retrieval pipeline. It is very useful — but it always ret...
Two different patterns There are two ways to use RAG inside an agent system. Both are valid — they just suit different s...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.