## Why This Project Exists Multi-agent systems are one of the most over-reached-for patterns in AI engineering. A team building an order-dispute resolution system for a Flipkart-style platform sees an agent framework with a multi-agent example, copies the pattern, and ends up with three coordinating agents doing what one well-designed agent with good tools could have done, at three times the latency and cost. This project takes the opposite approach: build the single-agent version first, honestly evaluate its limits, and only add a second agent when you can point to specific evidence that the single agent genuinely cannot handle the task. The deliverable is not just working code, it is a comparison document with real cost and latency numbers that either justifies the added complexity or proves it was unnecessary. ### The task: order dispute resolution A customer disputes a charge, claiming an item arrived damaged. Resolving this requires: reading the customer's complaint and photos, checking the order and delivery record, checking the seller's dispute history for a pattern of complaints, and deciding whether to refund, request more evidence, or escalate to a human. This genuinely spans multiple domains of "expertise," which makes it a fair test case for whether multi-agent coordination earns its cost here. ### Architecture diagram: what you will build and compare SINGLE AGENT VERSION MULTI-AGENT VERSION +----------------+ +----------------+ | One Agent | | Orchestrator | | 4 tools: | | Agent | | - read_dispute| +----------------+ | - check_order | | | | | - check_seller| v v v | - decide | +--------+ +--------+ +--------+ +----------------+ | Dispute| | Order | | Seller | | Reader | | Checker| | History| | Agent | | Agent | | Agent | +--------+ +--------+ +--------+ \ | / v v v +----------------+ | Decision Agent | +----------------+ > 📌 **Engineering Decision:** Multi-agent only once a single agent demonstrably can't handle the task. This project is built specifically to test that decision honestly rather than assume it, do not skip Milestone 1's single-agent version even if you are confident multi-agent will win, the comparison is worthless without a fair baseline. ### Prerequisites This project assumes you have completed Project 2 (Reliable AI Agent), you already know LangGraph and single-agent design from that project, this one extends the same skills into a coordination question. > 💡 **Tip:** Write down your prediction now, before building either version: do you expect multi-agent to win on quality, lose on cost and latency, or some mix? Milestone 4 asks you to compare your prediction against what you actually measured, and being honestly wrong is a more useful lesson than being vaguely right.
## Why This Milestone Cannot Be Skipped A multi-agent system that beats a strawman single agent proves nothing. The single agent needs the same tools, the same model, and a genuinely reasonable prompt, built with the same care as Project 2's refund agent, not a rushed placeholder designed to lose the comparison. ### Defining the tools ```python from pydantic import BaseModel from typing import Literal class DisputeDetails(BaseModel): dispute_id: str customer_claim: str has_photo_evidence: bool class OrderRecord(BaseModel): order_id: str delivery_status: Literal["delivered", "delivery_disputed", "in_transit"] delivery_photo_url: str | None class SellerHistory(BaseModel): seller_id: str dispute_rate_last_90_days: float total_orders_last_90_days: int class DisputeDecision(BaseModel): dispute_id: str decision: Literal["refund", "request_more_evidence", "escalate_to_human"] reasoning: str ``` ```python def read_dispute(dispute_id: str) -> DisputeDetails: """Fetch the customer's dispute claim and whether photos were attached.""" record = disputes_db.find_one({"dispute_id": dispute_id}) return DisputeDetails(**record) def check_order(order_id: str) -> OrderRecord: """Fetch delivery status and any delivery confirmation photo.""" order = orders_db.find_one({"order_id": order_id}) return OrderRecord(**order) def check_seller_history(seller_id: str) -> SellerHistory: """ Fetch the seller's recent dispute rate, a seller with an unusually high dispute rate is a signal worth weighing in the decision. """ history = sellers_db.find_one({"seller_id": seller_id}) return SellerHistory(**history) ``` ### The single agent's reasoning prompt ```python SINGLE_AGENT_SYSTEM_PROMPT = """You resolve order disputes for a Flipkart-style marketplace. You have access to three tools: read_dispute, check_order, and check_seller_history. Use them to gather what you need, then decide: refund, request_more_evidence, or escalate_to_human. Escalate to a human if the seller's dispute_rate_last_90_days exceeds 15%, or if the evidence is genuinely ambiguous. Otherwise decide directly.""" ``` > 🔴 **Common Mistake:** Building an intentionally weak single-agent baseline to make the multi-agent version look better by comparison. If the single agent's prompt is vague or its tools are poorly described, following the same tool-design discipline from Project 2, any comparison built on it is measuring a strawman, not a genuine architectural tradeoff. ### Guided practice Build the single agent in LangGraph following the same reason-act-observe pattern from Project 2. Run it against 15 test disputes covering clear-refund cases, clear-escalation cases, and ambiguous cases. Record the decision, total latency, and total token cost for each of the 15 runs, you will need these numbers in Milestone 4.
## What Actually Changes with Multiple Agents The multi-agent version splits the same work across specialized agents coordinated by an orchestrator. Each sub-agent focuses on one domain, dispute reading, order checking, or seller history, and the orchestrator combines their outputs into a final decision. This can help when each sub-task genuinely benefits from a distinct prompt, context, or even a different model, it can also just add coordination overhead for no benefit if the sub-tasks were simple enough for one agent all along. ### Defining the sub-agents ```python from langgraph.graph import StateGraph, END from typing import TypedDict class DisputeState(TypedDict): dispute_id: str dispute_details: dict | None order_record: dict | None seller_history: dict | None final_decision: dict | None def dispute_reader_agent(state: DisputeState) -> DisputeState: """ Specialized agent focused only on reading and summarizing the customer's claim and evidence quality. """ details = read_dispute(state["dispute_id"]) state["dispute_details"] = details.model_dump() return state def order_checker_agent(state: DisputeState) -> DisputeState: """Specialized agent focused only on delivery record verification.""" order_id = state["dispute_details"]["order_id"] record = check_order(order_id) state["order_record"] = record.model_dump() return state def seller_history_agent(state: DisputeState) -> DisputeState: """Specialized agent focused only on seller pattern analysis.""" seller_id = state["order_record"]["seller_id"] history = check_seller_history(seller_id) state["seller_history"] = history.model_dump() return state ``` ### The orchestrator and decision agent ```python def decision_agent(state: DisputeState) -> DisputeState: """ Combines outputs from all three sub-agents into a final decision. This is where the coordination actually pays off, or doesn't, depending on whether combining separately-gathered context genuinely produces a better decision than one agent gathering it directly. """ prompt = f"""Dispute: {state['dispute_details']} Order record: {state['order_record']} Seller history: {state['seller_history']} Decide: refund, request_more_evidence, or escalate_to_human.""" # LLM call using the combined context from all three agents decision = call_llm_for_decision(prompt) state["final_decision"] = decision return state graph = StateGraph(DisputeState) graph.add_node("read_dispute", dispute_reader_agent) graph.add_node("check_order", order_checker_agent) graph.add_node("check_seller", seller_history_agent) graph.add_node("decide", decision_agent) graph.set_entry_point("read_dispute") graph.add_edge("read_dispute", "check_order") graph.add_edge("check_order", "check_seller") graph.add_edge("check_seller", "decide") graph.add_edge("decide", END) multi_agent_app = graph.compile() ``` > **Note:** Notice this multi-agent graph is actually sequential, each agent hands off to the next rather than working in parallel. That is deliberate here since each step depends on the previous one's output (you need the order record before you can look up the seller). A genuinely parallel multi-agent design would only apply if the sub-tasks were independent of each other, which is worth noticing as you build this. > 🔴 **Common Mistake:** Assuming "multi-agent" automatically means "parallel and therefore faster." A sequential multi-agent pipeline like this one is not faster than a single agent doing the same steps, it is usually slower, since each agent hop adds its own LLM call overhead on top of the same total work. ### Concept check Before running any tests, predict: will this multi-agent version have higher or lower total token cost than the single agent from Milestone 1 for the same 15 disputes? Consider that each sub-agent may make its own LLM call to reason about its narrow task, versus one agent reasoning once with all three tools available.
## Building a Fair, Controlled Comparison A comparison is only meaningful if both versions face identical conditions: the same 15 disputes, the same underlying data, the same model, run back to back so external factors like API latency variance are minimized. ### The test harness ```python import time def run_comparison(dispute_ids: list[str]): results = {"single_agent": [], "multi_agent": []} for dispute_id in dispute_ids: ## Single agent run start = time.time() single_result = single_agent_app.invoke({"dispute_id": dispute_id, "step_count": 0}) single_latency = time.time() - start results["single_agent"].append({ "dispute_id": dispute_id, "decision": single_result["final_decision"], "latency_seconds": single_latency, "total_tokens": single_result.get("total_tokens", 0) }) ## Multi-agent run, same dispute, immediately after start = time.time() multi_result = multi_agent_app.invoke({"dispute_id": dispute_id}) multi_latency = time.time() - start results["multi_agent"].append({ "dispute_id": dispute_id, "decision": multi_result["final_decision"], "latency_seconds": multi_latency, "total_tokens": multi_result.get("total_tokens", 0) }) return results ``` > 💡 **Tip:** Track token usage per agent hop, not just the total, in the multi-agent version. If the orchestrator's final decision call ends up doing most of the actual reasoning work anyway, that is itself evidence the earlier sub-agents added coordination overhead without adding proportional value. ### Scoring decision quality Latency and cost are objective, decision quality needs a rubric. Since you already know the correct outcome for each of your 15 test disputes (you designed them), score each version's decision as correct or incorrect against your known-correct answer. ```python def score_decisions(results: dict, ground_truth: dict): for version in ["single_agent", "multi_agent"]: correct = 0 for r in results[version]: expected = ground_truth[r["dispute_id"]] if r["decision"]["decision"] == expected: correct += 1 accuracy = correct / len(results[version]) print(f"{version}: {accuracy:.0%} accuracy") ``` > 🔴 **Common Mistake:** Comparing only the final decision's correctness while ignoring cost and latency entirely. A multi-agent system that is 2% more accurate but costs 3 times as much and takes 4 times as long is not obviously the better system, that tradeoff needs to be stated explicitly, not buried. ### Guided practice Run `run_comparison` across all 15 test disputes for both versions. Produce a table with columns: dispute_id, single-agent decision, multi-agent decision, single-agent latency, multi-agent latency, single-agent tokens, multi-agent tokens. This raw table is the evidence base for Milestone 4's written comparison.
## Why the Document Is the Actual Deliverable Working code for both versions proves you can build multi-agent systems. The comparison document proves you know when to. This is what the capstone project spec asks for explicitly: a comparison doc proving a single agent could or couldn't handle the task, backed by real cost and latency data, not a general opinion about multi-agent architectures. ### Structuring the comparison Your document should answer four questions directly, using the numbers from Milestone 3, not general claims: 1. **Did the multi-agent version produce more accurate decisions?** State the exact accuracy percentage for each version, and specifically which disputes they disagreed on, not just the aggregate number. 2. **What did the accuracy difference cost in latency?** State the average and worst-case latency for each version. A multi-agent system with a 4-second average versus a single agent's 1.5-second average is a real user-facing cost, especially in a support-chat context where response time is visible. 3. **What did it cost in tokens and money?** State total tokens consumed across all 15 test disputes for each version, converted to an actual rupee cost estimate using current API pricing. This is the number that matters most at real production traffic volume. 4. **Was the added complexity worth it for this specific task?** Give a direct verdict, not a hedge. If the single agent matched multi-agent accuracy at a third of the cost, say so plainly. If multi-agent caught a category of dispute the single agent consistently missed, say exactly which category and why the specialization helped. ### Example verdict structure ```text Verdict: For this order-dispute task, the single agent matched multi-agent decision accuracy (13/15 vs 14/15 correct) at approximately one-third the token cost and half the latency. The one dispute the multi-agent version got right and the single agent missed involved a seller with a borderline dispute rate (14.8%, just under the 15% escalation threshold) where the seller-history sub-agent's isolated focus caught a pattern the single agent's combined reasoning under-weighted. Recommendation: Ship the single-agent version. The one edge case multi-agent caught does not justify 3x cost across all traffic. Revisit if seller-history edge cases become a larger share of real disputes over time. ``` > 📌 **Engineering Decision:** A specific, numbers-backed verdict, even if it is unflattering to the more complex system you spent more time building. The value of this project is in the honest comparison, not in justifying the extra work multi-agent took to build. If your data says single-agent wins, write that down clearly. > 🔴 **Common Mistake:** Writing a comparison document that hedges with "it depends on the use case" without ever stating what your specific use case's data actually showed. That sentence is true of every architectural decision in software and says nothing, the document needs your specific numbers and your specific verdict for this specific task. ### Capstone lab checkpoint Write the full comparison document using the four-question structure above, backed by your actual Milestone 3 data. Include the raw comparison table as an appendix. State one clear recommendation, and one condition under which you would revisit that recommendation later.
## Final Verification ### 1. Both versions run correctly ```bash python run_agent.py --version single --dispute-id DSP-4021 python run_agent.py --version multi --dispute-id DSP-4021 ``` Expected output: both versions return a valid `DisputeDecision` object with a `decision` field set to one of the three allowed values, no errors on either path. ### 2. Comparison data is complete Confirm your Milestone 3 comparison table has all 15 rows populated, with no missing latency or token values for either version, on any dispute. ### 3. Ground truth is genuinely known Before trusting your accuracy scoring, confirm each of your 15 test disputes has an unambiguous correct answer that you determined independently, not one you reverse-engineered from whichever version happened to answer first. ### 4. Comparison document has a clear verdict Read back your own Milestone 4 document. Confirm it states a specific recommendation (ship single-agent, ship multi-agent, or a defined hybrid) rather than only listing tradeoffs without concluding anything. ### Quick reference | Question | Single Agent | Multi-Agent | |:---|:---|:---| | Decision accuracy | Fill in from Milestone 3 | Fill in from Milestone 3 | | Average latency | Fill in from Milestone 3 | Fill in from Milestone 3 | | Total token cost | Fill in from Milestone 3 | Fill in from Milestone 3 | | Coordination overhead | None | Orchestrator + 3 sub-agent hops | ### Common mistakes across the full project Building an intentionally weak single-agent baseline to make multi-agent look better invalidates the entire comparison, and the fix is applying the same tool-design and prompt-quality discipline from Project 2 to both versions equally. Assuming multi-agent is automatically faster because it sounds parallel produces a false expectation, and the fix is checking whether your specific sub-agents' work is genuinely independent or, as in this project, sequential and therefore strictly slower. Comparing only final decision accuracy while ignoring cost and latency hides the real tradeoff, and the fix is reporting all three numbers together, never accuracy alone. Reaching for multi-agent before proving a single agent can't do it is the exact anti-pattern this project exists to test, and the fix is treating Milestone 1's honest baseline as mandatory, not a formality to rush past. Writing a hedged comparison document that avoids a concrete verdict wastes the entire data-gathering effort, and the fix is stating your specific numbers and your specific recommendation plainly, even when the answer disappoints the more complex system you spent more time building. > 💡 **Tip:** Whichever version you recommend shipping, keep both implementations in your portfolio repository. The comparison document itself, not just the winning code, is what demonstrates real engineering judgment to anyone reviewing your capstone.
Why This Project Exists Multi-agent systems are one of the most over-reached-for patterns in AI engineering. A team buil...
Why This Milestone Cannot Be Skipped A multi-agent system that beats a strawman single agent proves nothing. The single ...
What Actually Changes with Multiple Agents The multi-agent version splits the same work across specialized agents coordi...
Building a Fair, Controlled Comparison A comparison is only meaningful if both versions face identical conditions: the s...
Why the Document Is the Actual Deliverable Working code for both versions proves you can build multi-agent systems. The ...
Final Verification 1. Both versions run correctly Expected output: both versions return a valid DisputeDecision object w...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.