Prove When Multi-Agent Orchestration Actually Beats a Single Agent

Build a multi-agent order-dispute system and prove with cost and latency data whether it beats a single agent.

Domains & Technologies

Domains

MULTI-AGENTLANGGRAPHORCHESTRATIONCOST-ANALYSIS

Technologies

Blueprint Walkthrough

Architecture Overview & Problem Statement

Why This Project Exists

Multi-agent systems are one of the most over-reached-for patterns in AI engineering. A team building an order-dispute resolution system for a Flipkart-style platform sees an agent framework with a multi-agent example, copies the pattern, and ends up with three coordinating agents doing what one well-designed agent with good tools could have done, at three times the latency and cost.

This project takes the opposite approach: build the single-agent version first, honestly evaluate its limits, and only add a second agent when you can point to specific evidence that the single agent genuinely cannot handle the task. The deliverable is not just working code, it is a comparison document with real cost and latency numbers that either justifies the added complexity or proves it was unnecessary.

The task: order dispute resolution

A customer disputes a charge, claiming an item arrived damaged. Resolving this requires: reading the customer's complaint and photos, checking the order and delivery record, checking the seller's dispute history for a pattern of complaints, and deciding whether to refund, request more evidence, or escalate to a human. This genuinely spans multiple domains of "expertise," which makes it a fair test case for whether multi-agent coordination earns its cost here.

Architecture diagram: what you will build and compare

◈ DIAGRAM

SINGLE AGENT VERSION                MULTI-AGENT VERSION
 
+----------------+                  +----------------+
|  One Agent     |                  | Orchestrator   |
|  4 tools:      |                  | Agent          |
|  - read_dispute|                  +----------------+
|  - check_order |                     |     |     |
|  - check_seller|                     v     v     v
|  - decide      |              +--------+ +--------+ +--------+
+----------------+              | Dispute| | Order  | | Seller |
                                | Reader | | Checker| | History|
                                | Agent  | | Agent  | | Agent  |
                                +--------+ +--------+ +--------+
                                     \        |        /
                                      v        v       v
                                   +----------------+
                                   | Decision Agent |
                                   +----------------+

Engineering Decision
Multi-agent only once a single agent demonstrably can't handle the task. This project is built specifically to test that decision honestly rather than assume it, do not skip Milestone 1's single-agent version even if you are confident multi-agent will win, the comparison is worthless without a fair baseline.

Prerequisites

This project assumes you have completed Project 2 (Reliable AI Agent), you already know LangGraph and single-agent design from that project, this one extends the same skills into a coordination question.

Tip
Write down your prediction now, before building either version: do you expect multi-agent to win on quality, lose on cost and latency, or some mix? Milestone 4 asks you to compare your prediction against what you actually measured, and being honestly wrong is a more useful lesson than being vaguely right.

Milestone 1: Build the Honest Single-Agent Baseline

Why This Milestone Cannot Be Skipped

A multi-agent system that beats a strawman single agent proves nothing. The single agent needs the same tools, the same model, and a genuinely reasonable prompt, built with the same care as Project 2's refund agent, not a rushed placeholder designed to lose the comparison.

Defining the tools

PYTHON

from pydantic import BaseModel
from typing import Literal
 
class DisputeDetails(BaseModel):
    dispute_id: str
    customer_claim: str
    has_photo_evidence: bool
 
class OrderRecord(BaseModel):
    order_id: str
    delivery_status: Literal["delivered", "delivery_disputed", "in_transit"]
    delivery_photo_url: str | None
 
class SellerHistory(BaseModel):
    seller_id: str
    dispute_rate_last_90_days: float
    total_orders_last_90_days: int
 
class DisputeDecision(BaseModel):
    dispute_id: str
    decision: Literal["refund", "request_more_evidence", "escalate_to_human"]
    reasoning: str

PYTHON

def read_dispute(dispute_id: str) -> DisputeDetails:
    """Fetch the customer's dispute claim and whether photos were attached."""
    record = disputes_db.find_one({"dispute_id": dispute_id})
    return DisputeDetails(**record)
 
def check_order(order_id: str) -> OrderRecord:
    """Fetch delivery status and any delivery confirmation photo."""
    order = orders_db.find_one({"order_id": order_id})
    return OrderRecord(**order)
 
def check_seller_history(seller_id: str) -> SellerHistory:
    """
    Fetch the seller's recent dispute rate, a seller with an unusually
    high dispute rate is a signal worth weighing in the decision.
    """
    history = sellers_db.find_one({"seller_id": seller_id})
    return SellerHistory(**history)

The single agent's reasoning prompt

PYTHON

SINGLE_AGENT_SYSTEM_PROMPT = """You resolve order disputes for a Flipkart-style
marketplace. You have access to three tools: read_dispute, check_order, and
check_seller_history. Use them to gather what you need, then decide:
refund, request_more_evidence, or escalate_to_human.
 
Escalate to a human if the seller's dispute_rate_last_90_days exceeds 15%,
or if the evidence is genuinely ambiguous. Otherwise decide directly."""

Common Mistake
Building an intentionally weak single-agent baseline to make the multi-agent version look better by comparison. If the single agent's prompt is vague or its tools are poorly described, following the same tool-design discipline from Project 2, any comparison built on it is measuring a strawman, not a genuine architectural tradeoff.

Guided practice

Build the single agent in LangGraph following the same reason-act-observe pattern from Project 2. Run it against 15 test disputes covering clear-refund cases, clear-escalation cases, and ambiguous cases. Record the decision, total latency, and total token cost for each of the 15 runs, you will need these numbers in Milestone 4.

Milestone 2: Build the Multi-Agent Version

What Actually Changes with Multiple Agents

The multi-agent version splits the same work across specialized agents coordinated by an orchestrator. Each sub-agent focuses on one domain, dispute reading, order checking, or seller history, and the orchestrator combines their outputs into a final decision. This can help when each sub-task genuinely benefits from a distinct prompt, context, or even a different model, it can also just add coordination overhead for no benefit if the sub-tasks were simple enough for one agent all along.

Defining the sub-agents

PYTHON

from langgraph.graph import StateGraph, END
from typing import TypedDict
 
class DisputeState(TypedDict):
    dispute_id: str
    dispute_details: dict | None
    order_record: dict | None
    seller_history: dict | None
    final_decision: dict | None
 
def dispute_reader_agent(state: DisputeState) -> DisputeState:
    """
    Specialized agent focused only on reading and summarizing
    the customer's claim and evidence quality.
    """
    details = read_dispute(state["dispute_id"])
    state["dispute_details"] = details.model_dump()
    return state
 
def order_checker_agent(state: DisputeState) -> DisputeState:
    """Specialized agent focused only on delivery record verification."""
    order_id = state["dispute_details"]["order_id"]
    record = check_order(order_id)
    state["order_record"] = record.model_dump()
    return state
 
def seller_history_agent(state: DisputeState) -> DisputeState:
    """Specialized agent focused only on seller pattern analysis."""
    seller_id = state["order_record"]["seller_id"]
    history = check_seller_history(seller_id)
    state["seller_history"] = history.model_dump()
    return state

The orchestrator and decision agent

PYTHON

def decision_agent(state: DisputeState) -> DisputeState:
    """
    Combines outputs from all three sub-agents into a final decision.
    This is where the coordination actually pays off, or doesn't,
    depending on whether combining separately-gathered context genuinely
    produces a better decision than one agent gathering it directly.
    """
    prompt = f"""Dispute: {state['dispute_details']}
    Order record: {state['order_record']}
    Seller history: {state['seller_history']}
 
    Decide: refund, request_more_evidence, or escalate_to_human."""
    # LLM call using the combined context from all three agents
    decision = call_llm_for_decision(prompt)
    state["final_decision"] = decision
    return state
 
graph = StateGraph(DisputeState)
graph.add_node("read_dispute", dispute_reader_agent)
graph.add_node("check_order", order_checker_agent)
graph.add_node("check_seller", seller_history_agent)
graph.add_node("decide", decision_agent)
 
graph.set_entry_point("read_dispute")
graph.add_edge("read_dispute", "check_order")
graph.add_edge("check_order", "check_seller")
graph.add_edge("check_seller", "decide")
graph.add_edge("decide", END)
 
multi_agent_app = graph.compile()

Note
Notice this multi-agent graph is actually sequential, each agent hands off to the next rather than working in parallel. That is deliberate here since each step depends on the previous one's output (you need the order record before you can look up the seller). A genuinely parallel multi-agent design would only apply if the sub-tasks were independent of each other, which is worth noticing as you build this.

Common Mistake
Assuming "multi-agent" automatically means "parallel and therefore faster." A sequential multi-agent pipeline like this one is not faster than a single agent doing the same steps, it is usually slower, since each agent hop adds its own LLM call overhead on top of the same total work.

Concept check

Before running any tests, predict: will this multi-agent version have higher or lower total token cost than the single agent from Milestone 1 for the same 15 disputes? Consider that each sub-agent may make its own LLM call to reason about its narrow task, versus one agent reasoning once with all three tools available.

Milestone 3: Run Both Versions on the Same Test Set

Building a Fair, Controlled Comparison

A comparison is only meaningful if both versions face identical conditions: the same 15 disputes, the same underlying data, the same model, run back to back so external factors like API latency variance are minimized.

The test harness

PYTHON

import time
 
def run_comparison(dispute_ids: list[str]):
    results = {"single_agent": [], "multi_agent": []}
 
    for dispute_id in dispute_ids:
        ## Single agent run
        start = time.time()
        single_result = single_agent_app.invoke({"dispute_id": dispute_id, "step_count": 0})
        single_latency = time.time() - start
        results["single_agent"].append({
            "dispute_id": dispute_id,
            "decision": single_result["final_decision"],
            "latency_seconds": single_latency,
            "total_tokens": single_result.get("total_tokens", 0)
        })
 
        ## Multi-agent run, same dispute, immediately after
        start = time.time()
        multi_result = multi_agent_app.invoke({"dispute_id": dispute_id})
        multi_latency = time.time() - start
        results["multi_agent"].append({
            "dispute_id": dispute_id,
            "decision": multi_result["final_decision"],
            "latency_seconds": multi_latency,
            "total_tokens": multi_result.get("total_tokens", 0)
        })
 
    return results

Tip
Track token usage per agent hop, not just the total, in the multi-agent version. If the orchestrator's final decision call ends up doing most of the actual reasoning work anyway, that is itself evidence the earlier sub-agents added coordination overhead without adding proportional value.

Scoring decision quality

Latency and cost are objective, decision quality needs a rubric. Since you already know the correct outcome for each of your 15 test disputes (you designed them), score each version's decision as correct or incorrect against your known-correct answer.

PYTHON

def score_decisions(results: dict, ground_truth: dict):
    for version in ["single_agent", "multi_agent"]:
        correct = 0
        for r in results[version]:
            expected = ground_truth[r["dispute_id"]]
            if r["decision"]["decision"] == expected:
                correct += 1
        accuracy = correct / len(results[version])
        print(f"{version}: {accuracy:.0%} accuracy")

Common Mistake
Comparing only the final decision's correctness while ignoring cost and latency entirely. A multi-agent system that is 2% more accurate but costs 3 times as much and takes 4 times as long is not obviously the better system, that tradeoff needs to be stated explicitly, not buried.

Guided practice

Run run_comparison across all 15 test disputes for both versions. Produce a table with columns: dispute_id, single-agent decision, multi-agent decision, single-agent latency, multi-agent latency, single-agent tokens, multi-agent tokens. This raw table is the evidence base for Milestone 4's written comparison.

Milestone 4: Write the Comparison Document

Why the Document Is the Actual Deliverable

Working code for both versions proves you can build multi-agent systems. The comparison document proves you know when to. This is what the capstone project spec asks for explicitly: a comparison doc proving a single agent could or couldn't handle the task, backed by real cost and latency data, not a general opinion about multi-agent architectures.

Structuring the comparison

Your document should answer four questions directly, using the numbers from Milestone 3, not general claims:

Did the multi-agent version produce more accurate decisions? State the exact accuracy percentage for each version, and specifically which disputes they disagreed on, not just the aggregate number.
What did the accuracy difference cost in latency? State the average and worst-case latency for each version. A multi-agent system with a 4-second average versus a single agent's 1.5-second average is a real user-facing cost, especially in a support-chat context where response time is visible.
What did it cost in tokens and money? State total tokens consumed across all 15 test disputes for each version, converted to an actual rupee cost estimate using current API pricing. This is the number that matters most at real production traffic volume.
Was the added complexity worth it for this specific task? Give a direct verdict, not a hedge. If the single agent matched multi-agent accuracy at a third of the cost, say so plainly. If multi-agent caught a category of dispute the single agent consistently missed, say exactly which category and why the specialization helped.

Example verdict structure

TEXT

Verdict: For this order-dispute task, the single agent matched multi-agent
decision accuracy (13/15 vs 14/15 correct) at approximately one-third the
token cost and half the latency. The one dispute the multi-agent version
got right and the single agent missed involved a seller with a borderline
dispute rate (14.8%, just under the 15% escalation threshold) where the
seller-history sub-agent's isolated focus caught a pattern the single
agent's combined reasoning under-weighted.
 
Recommendation: Ship the single-agent version. The one edge case multi-agent
caught does not justify 3x cost across all traffic. Revisit if seller-history
edge cases become a larger share of real disputes over time.

Engineering Decision
A specific, numbers-backed verdict, even if it is unflattering to the more complex system you spent more time building. The value of this project is in the honest comparison, not in justifying the extra work multi-agent took to build. If your data says single-agent wins, write that down clearly.

Common Mistake
Writing a comparison document that hedges with "it depends on the use case" without ever stating what your specific use case's data actually showed. That sentence is true of every architectural decision in software and says nothing, the document needs your specific numbers and your specific verdict for this specific task.

Capstone lab checkpoint

Write the full comparison document using the four-question structure above, backed by your actual Milestone 3 data. Include the raw comparison table as an appendix. State one clear recommendation, and one condition under which you would revisit that recommendation later.

Validation & Testing

Final Verification

1. Both versions run correctly

Bash

python run_agent.py --version single --dispute-id DSP-4021
python run_agent.py --version multi --dispute-id DSP-4021

Expected output: both versions return a valid DisputeDecision object with a decision field set to one of the three allowed values, no errors on either path.

2. Comparison data is complete

Confirm your Milestone 3 comparison table has all 15 rows populated, with no missing latency or token values for either version, on any dispute.

3. Ground truth is genuinely known

Before trusting your accuracy scoring, confirm each of your 15 test disputes has an unambiguous correct answer that you determined independently, not one you reverse-engineered from whichever version happened to answer first.

4. Comparison document has a clear verdict

Read back your own Milestone 4 document. Confirm it states a specific recommendation (ship single-agent, ship multi-agent, or a defined hybrid) rather than only listing tradeoffs without concluding anything.

Quick reference

Question	Single Agent	Multi-Agent
Decision accuracy	Fill in from Milestone 3	Fill in from Milestone 3
Average latency	Fill in from Milestone 3	Fill in from Milestone 3
Total token cost	Fill in from Milestone 3	Fill in from Milestone 3
Coordination overhead	None	Orchestrator + 3 sub-agent hops

Common mistakes across the full project

Building an intentionally weak single-agent baseline to make multi-agent look better invalidates the entire comparison, and the fix is applying the same tool-design and prompt-quality discipline from Project 2 to both versions equally. Assuming multi-agent is automatically faster because it sounds parallel produces a false expectation, and the fix is checking whether your specific sub-agents' work is genuinely independent or, as in this project, sequential and therefore strictly slower. Comparing only final decision accuracy while ignoring cost and latency hides the real tradeoff, and the fix is reporting all three numbers together, never accuracy alone. Reaching for multi-agent before proving a single agent can't do it is the exact anti-pattern this project exists to test, and the fix is treating Milestone 1's honest baseline as mandatory, not a formality to rush past. Writing a hedged comparison document that avoids a concrete verdict wastes the entire data-gathering effort, and the fix is stating your specific numbers and your specific recommendation plainly, even when the answer disappoints the more complex system you spent more time building.

Tip
Whichever version you recommend shipping, keep both implementations in your portfolio repository. The comparison document itself, not just the winning code, is what demonstrates real engineering judgment to anyone reviewing your capstone.

Videos & Guides

DOCUMENTATION

LangGraph Multi-Agent Patterns

Official LangGraph documentation on multi-agent coordination patterns and when they apply.

View Reference