Multi-Agent vs Single Agent: Cost and Latency Test

Build a single agent and a LangGraph multi-agent system for order disputes, then measure accuracy, latency and cost to decide which one to ship.

Architecture Overview & Problem Statement

This project builds the same order-dispute resolver twice, once as a single tool-using agent and once as a LangGraph multi-agent system with specialist agents and a decision agent. It then runs both on an identical test set and measures decision accuracy, latency and token cost, so you can decide with data which design to ship. The deliverable is not only working code. It is a numbers-backed verdict on whether the extra coordination was worth paying for.

Multi-agent designs are one of the most over-used patterns in AI engineering. A team building dispute handling for a marketplace like Flipkart sees a framework example with three cooperating agents, copies it, and ends up paying several times the latency and cost of one well-designed agent that had the same tools. This project tests that choice honestly. Each version gets the same model, the same policy text and the same data, and you also stress the baseline with noisy tool output to find out whether there is a point where specialisation starts to pay off.

What you build, learn and can show

You build: a fixed 15-case test set with known answers, a single-agent baseline, a multi-agent graph with parallel specialists, a benchmark harness that records accuracy, latency and tokens, and a noise knob that simulates verbose real-world tool output.
You learn: how tool-calling loops and graph orchestration differ in LLM calls and latency, how to measure token cost correctly, how to run a fair comparison, and how to write a verdict that commits to a recommendation.
You can show: a repository with both implementations, raw CSV results, a summary table, and a one-page decision document. That document is what proves engineering judgement.

The task: order dispute resolution

A customer disputes a charge because an item arrived damaged. Resolving it means reading the claim, checking the order and delivery record, and checking the seller's recent dispute rate, then choosing one of three outcomes: refund, request_more_evidence or escalate_to_human. The decision follows a fixed policy, which is what lets you score answers objectively:

Order	Rule	Outcome
1	Seller dispute rate over the last 90 days is above 0.15	`escalate_to_human`
2	Order value is above 20,000 INR	`escalate_to_human`
3	Delivery status is not `delivered`	`escalate_to_human`
4	The customer does not say the item itself is damaged (for example only the box)	`request_more_evidence`
5	No photo evidence was attached	`request_more_evidence`
6	Otherwise	`refund`

The two designs

The single agent has three tools and the whole policy. It reads the dispute, then checks the order and the seller, then decides.

flowchart TD D([Dispute ID]) --> A[One agent
3 tools + policy] A -->|tool calls| T[(Disputes, orders,
sellers data)] A --> R([Decision]) classDef start fill:#EDE9FE,color:#1E293B classDef agent fill:#326CE5,color:#FFFFFF classDef data fill:#10B981,color:#FFFFFF class D,R start class A agent class T data

The multi-agent version splits the work. The dispute record already contains the order ID and seller ID, so the order check and the seller check do not depend on each other and can run in parallel. That gives the multi-agent design a fair chance to recover latency.

flowchart TD D([Dispute ID]) --> RD[Dispute reader] RD --> OC[Order checker] RD --> SA[Seller analyst] OC --> DE[Decision agent] SA --> DE DE --> R([Decision]) classDef start fill:#EDE9FE,color:#1E293B classDef agent fill:#326CE5,color:#FFFFFF classDef decide fill:#F97316,color:#FFFFFF class D,R start class RD,OC,SA agent class DE decide

Prerequisites

Python 3.11 or newer, an Anthropic API key, and basic familiarity with tool calling. Building a guarded single agent first, as in the Reliable AI Agent project, helps but is not required.
A small budget. The runs below use a compact model and a few hundred short calls, but check current pricing before you start and set a spending limit on your key.

Tip
Write down your prediction before you build anything: which version wins on accuracy, which on cost, and which on latency? Milestone 6 asks you to compare it with what you measured. Being wrong with evidence teaches more than being vaguely right.

Milestone 1: Set Up the Project and the Fixed Test Set

This milestone creates the project, the 15 test cases with known answers and the three tools. A comparison is only trustworthy if both versions face exactly the same data, so everything here is deterministic and lives in code.

Bash

mkdir agent-comparison && cd agent-comparison
python3 -m venv .venv && source .venv/bin/activate
pip install langgraph langchain-anthropic langchain-core pandas
pip freeze | grep -iE "langgraph|langchain|pandas|anthropic" > requirements.txt
 
export ANTHROPIC_API_KEY=your-key-here
export MODEL=claude-haiku-5-5

Security
Never commit your API key. Keep it in the environment or a git-ignored .env file, and set a monthly spend limit on the key before running benchmarks that loop.

Library APIs in this area move quickly, so pin the versions you tested with. If an import fails later, check the changelog for the renamed function rather than rewriting your design.

Create cases.py. Each row is one dispute with the answer determined by applying the policy table above, not by running either agent:

Python

## id, customer claim, photo attached, delivery status, order value INR, seller dispute rate, expected decision
CASES = [
    ("DSP-001", "The phone screen arrived cracked.", True, "delivered", 8999, 0.04, "refund"),
    ("DSP-002", "The blender jar is broken and leaks.", True, "delivered", 3499, 0.06, "refund"),
    ("DSP-003", "The shoe sole came off on day one.", True, "delivered", 2199, 0.02, "refund"),
    ("DSP-004", "The headphones are dead on arrival.", True, "delivered", 4599, 0.09, "refund"),
    ("DSP-005", "The outer box was dented but the product works fine.", True, "delivered", 5299, 0.11, "request_more_evidence"),
    ("DSP-006", "The watch glass is shattered.", True, "delivered", 6999, 0.151, "escalate_to_human"),
    ("DSP-007", "The laptop bag zip is broken.", True, "delivered", 1799, 0.22, "escalate_to_human"),
    ("DSP-008", "The table lamp arrived damaged.", True, "delivered", 2599, 0.148, "refund"),
    ("DSP-009", "The speaker has a crack along the side.", True, "delivered", 3999, 0.15, "refund"),
    ("DSP-010", "The camera lens arrived smashed.", True, "delivered", 48999, 0.05, "escalate_to_human"),
    ("DSP-011", "The tablet screen is cracked.", True, "delivered", 24999, 0.03, "escalate_to_human"),
    ("DSP-012", "My kettle arrived damaged.", False, "delivered", 1499, 0.05, "request_more_evidence"),
    ("DSP-013", "The chair leg is broken.", False, "delivered", 3299, 0.07, "request_more_evidence"),
    ("DSP-014", "My item arrived damaged.", True, "in_transit", 2999, 0.04, "escalate_to_human"),
    ("DSP-015", "The jacket zip is torn and the fabric ripped.", True, "delivery_disputed", 3599, 0.06, "escalate_to_human"),
]
 
DISPUTES, ORDERS, SELLERS, EXPECTED = {}, {}, {}, {}
for i, (did, claim, photo, delivery, value, rate, expected) in enumerate(CASES, start=1):
    oid, sid = f"ORD-{i:03d}", f"SEL-{i:03d}"
    DISPUTES[did] = {
        "dispute_id": did,
        "customer_claim": claim,
        "has_photo_evidence": photo,
        "order_id": oid,
        "seller_id": sid,
    }
    ORDERS[oid] = {"order_id": oid, "delivery_status": delivery, "order_value_inr": value}
    SELLERS[sid] = {
        "seller_id": sid,
        "dispute_rate_last_90_days": rate,
        "total_orders_last_90_days": 1200,
    }
    EXPECTED[did] = expected

Notice the deliberate edge cases. DSP-008 (0.148) and DSP-009 (exactly 0.15) sit at or just under the seller threshold and must be refunded, while DSP-006 (0.151) must escalate. DSP-005 tests whether an agent actually reads the claim.

Now tools.py. The NOISE setting pads each tool result with irrelevant records, which simulates the verbose audit logs real systems return. It stays at zero until Milestone 5:

Python

import json
import os
import random
 
from langchain_core.tools import tool
 
from cases import DISPUTES, ORDERS, SELLERS
 
NOISE = int(os.getenv("NOISE", "0"))
 
NOTES = [
    "Routine restock confirmed by warehouse",
    "Invoice reprinted at customer request",
    "Delivery address verified by courier",
    "Promotional email sent to buyer",
    "Packaging weight recorded at dispatch",
    "Loyalty points credited after purchase",
]
 
 
def _filler(kind: str, count: int) -> list[dict]:
    rng = random.Random(f"{kind}-{count}")
    return [
        {"ref": f"{kind}-{i}", "note": rng.choice(NOTES), "amount_inr": rng.randint(100, 9000)}
        for i in range(count)
    ]
 
 
@tool
def read_dispute(dispute_id: str) -> str:
    """Return the customer's claim, whether photos were attached, and the linked order_id and seller_id."""
    record = {**DISPUTES[dispute_id], "audit_log": _filler("dispute", NOISE)}
    return json.dumps(record)
 
 
@tool
def check_order(order_id: str) -> str:
    """Return the order's delivery_status (delivered, delivery_disputed or in_transit) and order_value_inr."""
    record = {**ORDERS[order_id], "audit_log": _filler("order", NOISE)}
    return json.dumps(record)
 
 
@tool
def check_seller_history(seller_id: str) -> str:
    """Return the seller's dispute_rate_last_90_days (0.0 to 1.0) and total_orders_last_90_days."""
    record = {**SELLERS[seller_id], "recent_events": _filler("seller", NOISE)}
    return json.dumps(record)

Create common.py with the shared model factory, the policy text and the output parser. Both versions import the same policy and the same model, which is what keeps the comparison fair:

Python

import os
import re
 
from langchain_anthropic import ChatAnthropic
 
MODEL = os.getenv("MODEL", "claude-haiku-5-5")
 
POLICY = """Dispute resolution policy. Apply the first rule that matches, in this order:
1. Seller dispute_rate_last_90_days is above 0.15: escalate_to_human.
2. order_value_inr is above 20000: escalate_to_human.
3. delivery_status is not 'delivered': escalate_to_human.
4. The customer does not say the item itself is damaged or defective (for example only the outer box is damaged): request_more_evidence.
5. has_photo_evidence is false: request_more_evidence.
6. Otherwise: refund.
End your final message with exactly one line: DECISION: <refund|request_more_evidence|escalate_to_human>"""
 
 
def make_llm() -> ChatAnthropic:
    return ChatAnthropic(model=MODEL, temperature=0, max_tokens=600)
 
 
def text_of(message) -> str:
    """Anthropic messages can hold a list of content blocks, so flatten to plain text."""
    content = message.content
    if isinstance(content, str):
        return content
    return "".join(block.get("text", "") for block in content if isinstance(block, dict))
 
 
def parse_decision(text: str) -> str:
    matches = re.findall(r"DECISION:\s*(refund|request_more_evidence|escalate_to_human)", text)
    return matches[-1] if matches else "unparseable"

Common Mistake
Putting the expected answer anywhere a model can see it, including in test descriptions or tool output. Keep EXPECTED out of the tools and prompts so you are measuring reasoning, not lookup.

Milestone 2: Build the Single-Agent Baseline

This milestone builds the honest baseline: one agent, three tools and the full policy. A multi-agent system that beats a deliberately weak baseline proves nothing, so give this version the same care you would give a real product.

The agent runs a reason-and-act loop. It calls read_dispute first because it needs the order and seller IDs, then can call check_order and check_seller_history together, then writes its decision. Save as single_agent.py:

Python

from langgraph.prebuilt import create_react_agent
 
from common import POLICY, make_llm, parse_decision, text_of
from tools import check_order, check_seller_history, read_dispute
 
SYSTEM = (
    "You resolve order disputes for an online marketplace. "
    "Use the tools to gather the facts you need, then decide.\n\n" + POLICY
)
 
agent = create_react_agent(
    make_llm(),
    [read_dispute, check_order, check_seller_history],
    prompt=SYSTEM,
)
 
 
def run_single(dispute_id: str) -> dict:
    result = agent.invoke({"messages": [("user", f"Resolve dispute {dispute_id}.")]})
    text = text_of(result["messages"][-1])
    return {"decision": parse_decision(text), "extra": {"final_text": text}}

Try it on one case. Expected output for DSP-001 is refund:

Bash

python -c "from single_agent import run_single; print(run_single('DSP-001'))"

Run it on two more cases that you know the answer to, such as DSP-006 (escalate) and DSP-012 (request more evidence), before you move on.

Common Mistake
Making the baseline weak by leaving the tool descriptions vague or the policy out of the prompt. Tool docstrings are what the model reads to decide which tool to call, so write them as carefully as you would for a production agent.

Milestone 3: Build the Multi-Agent Version

This milestone builds the same capability as a LangGraph graph of specialist agents. Each specialist has one tool and one narrow job, and a decision agent combines their written findings. This design can help when different subtasks benefit from different prompts or models. It can also just add calls for nothing, which is what you are here to find out.

The shared state holds each agent's finding. The order checker and seller analyst write to different keys, so they can run in parallel without conflicts. Save as multi_agent.py:

Python

from typing import TypedDict
 
from langgraph.graph import END, START, StateGraph
from langgraph.prebuilt import create_react_agent
 
from common import POLICY, make_llm, parse_decision, text_of
from tools import check_order, check_seller_history, read_dispute
 
 
class State(TypedDict, total=False):
    dispute_id: str
    dispute_finding: str
    order_finding: str
    seller_finding: str
    answer: str
 
 
FACTS_ONLY = "Report facts only. Do not make a decision. Keep the report under 60 words."
 
reader = create_react_agent(
    make_llm(),
    [read_dispute],
    prompt=(
        "You are the dispute reader. Fetch the dispute. Report the customer's claim, "
        "whether the customer says the item itself is damaged or defective, whether photo "
        "evidence was attached, the order_id and the seller_id. " + FACTS_ONLY
    ),
)
order_checker = create_react_agent(
    make_llm(),
    [check_order],
    prompt="You are the order checker. Look up the order named in the dispute finding. "
    "Report delivery_status and order_value_inr. " + FACTS_ONLY,
)
seller_analyst = create_react_agent(
    make_llm(),
    [check_seller_history],
    prompt="You are the seller analyst. Look up the seller named in the dispute finding. "
    "Report dispute_rate_last_90_days and total_orders_last_90_days. " + FACTS_ONLY,
)
decider_llm = make_llm()
 
 
def _run(agent, prompt: str) -> str:
    result = agent.invoke({"messages": [("user", prompt)]})
    return text_of(result["messages"][-1])
 
 
def read_node(state: State) -> State:
    return {"dispute_finding": _run(reader, f"Fetch dispute {state['dispute_id']}.")}
 
 
def order_node(state: State) -> State:
    prompt = f"Dispute finding:\n{state['dispute_finding']}\nLook up the order."
    return {"order_finding": _run(order_checker, prompt)}
 
 
def seller_node(state: State) -> State:
    prompt = f"Dispute finding:\n{state['dispute_finding']}\nLook up the seller."
    return {"seller_finding": _run(seller_analyst, prompt)}
 
 
def decide_node(state: State) -> State:
    reply = decider_llm.invoke(
        [
            ("system", "You make the final decision on an order dispute.\n\n" + POLICY),
            (
                "user",
                f"Dispute finding:\n{state['dispute_finding']}\n\n"
                f"Order finding:\n{state['order_finding']}\n\n"
                f"Seller finding:\n{state['seller_finding']}",
            ),
        ]
    )
    return {"answer": text_of(reply)}
 
 
graph = StateGraph(State)
graph.add_node("read", read_node)
graph.add_node("order", order_node)
graph.add_node("seller", seller_node)
graph.add_node("decide", decide_node)
graph.add_edge(START, "read")
graph.add_edge("read", "order")
graph.add_edge("read", "seller")
graph.add_edge("order", "decide")
graph.add_edge("seller", "decide")
graph.add_edge("decide", END)
multi_app = graph.compile()
 
 
def run_multi(dispute_id: str) -> dict:
    state = multi_app.invoke({"dispute_id": dispute_id})
    return {
        "decision": parse_decision(state["answer"]),
        "extra": {
            "dispute_finding": state["dispute_finding"],
            "order_finding": state["order_finding"],
            "seller_finding": state["seller_finding"],
        },
    }

Run the same three cases as before and compare the answers. Keep the findings in extra, because when the multi-agent version is wrong, the first place to look is whether a specialist dropped a fact the decider needed.

Bash

python -c "from multi_agent import run_multi; print(run_multi('DSP-006'))"

Count the language-model calls each design makes for one dispute. The single agent usually needs about three: read, check both in parallel, decide. The multi-agent graph needs about seven, because every specialist spends one call choosing its tool and one summarising the result, plus the decider. The parallel branch hides some of that in wall-clock time but not in tokens or money.

Common Mistake
Assuming multi-agent means faster because it sounds parallel. Only the order and seller branches here run side by side. The reader and decider are sequential, so the critical path is still longer than the single agent's, and the total work is larger.

Milestone 4: Measure Both Versions Fairly

This milestone builds the benchmark harness. A fair measurement uses identical cases, the same model, repeated runs to smooth out randomness, and a randomised run order so that neither version always benefits from warm connections or quieter API moments.

Token usage is captured with LangChain's usage callback, which counts every model call made inside the block, including those inside sub-agents. Prices are read from your environment so you always use the current numbers from your provider's pricing page rather than numbers baked into code.

Save as bench.py:

Python

import json
import os
import random
import time
 
import pandas as pd
from langchain_core.callbacks import get_usage_metadata_callback
 
from cases import EXPECTED
from multi_agent import run_multi
from single_agent import run_single
 
RUNS = int(os.getenv("RUNS", "3"))
NOISE = int(os.getenv("NOISE", "0"))
PRICE_IN = float(os.getenv("PRICE_IN_PER_MTOK", "0"))    ## USD per million input tokens
PRICE_OUT = float(os.getenv("PRICE_OUT_PER_MTOK", "0"))  ## USD per million output tokens
 
VERSIONS = {"single": run_single, "multi": run_multi}
 
 
def measure(name: str, fn, dispute_id: str) -> dict:
    with get_usage_metadata_callback() as usage:
        start = time.perf_counter()
        outcome = fn(dispute_id)
        latency = time.perf_counter() - start
    tokens_in = sum(u["input_tokens"] for u in usage.usage_metadata.values())
    tokens_out = sum(u["output_tokens"] for u in usage.usage_metadata.values())
    assert tokens_in + tokens_out > 0, "usage callback captured nothing, check library versions"
    cost = tokens_in / 1e6 * PRICE_IN + tokens_out / 1e6 * PRICE_OUT
    return {
        "version": name,
        "dispute_id": dispute_id,
        "decision": outcome["decision"],
        "expected": EXPECTED[dispute_id],
        "correct": outcome["decision"] == EXPECTED[dispute_id],
        "latency_s": latency,
        "tokens_in": tokens_in,
        "tokens_out": tokens_out,
        "cost_usd": cost,
        "extra": json.dumps(outcome["extra"]),
    }
 
 
def main() -> None:
    rng = random.Random(7)
    rows = []
    for repeat in range(RUNS):
        for dispute_id in EXPECTED:
            order = list(VERSIONS)
            rng.shuffle(order)  ## randomise which version runs first on each case
            for name in order:
                rows.append({**measure(name, VERSIONS[name], dispute_id), "repeat": repeat})
        print(f"finished repeat {repeat + 1} of {RUNS}")
 
    df = pd.DataFrame(rows)
    out_file = f"results-noise{NOISE}.csv"
    df.to_csv(out_file, index=False)
 
    grouped = df.groupby("version")
    summary = pd.DataFrame(
        {
            "accuracy": grouped["correct"].mean(),
            "latency_mean_s": grouped["latency_s"].mean(),
            "latency_p95_s": grouped["latency_s"].quantile(0.95),
            "tokens_in_mean": grouped["tokens_in"].mean(),
            "tokens_out_mean": grouped["tokens_out"].mean(),
            "usd_per_1000_disputes": grouped["cost_usd"].mean() * 1000,
        }
    )
    print(summary.round(3).to_string())
 
    wrong = df[~df["correct"]].groupby(["dispute_id", "version"]).size()
    print("\nWrong answers by case and version:")
    print(wrong.to_string() if len(wrong) else "none")
    print(f"\nraw rows saved to {out_file}")
 
 
if __name__ == "__main__":
    main()

Set the prices from your provider's current pricing page, run a single repeat first to catch errors cheaply, then the full run:

Bash

export PRICE_IN_PER_MTOK=0.0    ## replace with the current input price
export PRICE_OUT_PER_MTOK=0.0   ## replace with the current output price
 
RUNS=1 python bench.py
RUNS=3 python bench.py

The summary prints one row per version. Keep results-noise0.csv, since it is the baseline evidence for the document you write in Milestone 6.

Tip
Look at the per-case wrong-answer list before the aggregate numbers. Two versions can both score 93% while failing on completely different cases, and that difference is the interesting part of the story.

Common Mistake
Comparing accuracy alone. A version that is 2 points more accurate but costs three times as much and takes twice as long is not obviously better, so report all three measures together and state the trade-off explicitly.

Milestone 5: Stress the Baseline with Noisy Data

On clean, small tool output, specialisation rarely pays, because there is little for one agent to be distracted by. Real systems return long audit logs. This milestone raises the noise to find out whether a single agent's growing context hurts it, and whether the specialists, which each see only their own tool output, keep their decider's context small.

Re-run the full benchmark with more filler in every tool result. Each level writes its own results file:

Bash

NOISE=100 RUNS=3 python bench.py
NOISE=400 RUNS=3 python bench.py

Combine the three result files and compare how each version changes as the data gets noisier:

Python

import glob
import pandas as pd
 
frames = []
for path in sorted(glob.glob("results-noise*.csv")):
    frame = pd.read_csv(path)
    frame["noise"] = int(path.split("noise")[1].split(".")[0])
    frames.append(frame)
df = pd.concat(frames)
 
table = df.groupby(["noise", "version"]).agg(
    accuracy=("correct", "mean"),
    latency_mean_s=("latency_s", "mean"),
    tokens_in_mean=("tokens_in", "mean"),
    cost_usd_per_1000=("cost_usd", lambda s: s.mean() * 1000),
)
print(table.round(3).to_string())

Read the table for a crossover. The questions to ask are specific:

Does either version's accuracy fall as noise grows, and on which cases?
Does the single agent's input token count grow faster than the multi-agent total? In the single agent every later call re-sends all earlier tool output, while each specialist reads only its own.
Is there a noise level where multi-agent cost per 1,000 disputes drops below the single agent's, or where its accuracy pulls ahead?
If a specialist is wrong, did it summarise away a fact the decider needed? Check the extra column for that row.

Do not assume the answer. It may be that the single agent stays ahead at every level, and that is a legitimate and useful result. The point is to know where, if anywhere, the line is for your data.

Common Mistake
Raising the noise until something flips and then reporting that level as typical. Use noise sizes that resemble your real tool output, and say so in the document. A crossover that only appears at unrealistic sizes is a curiosity, not a reason to ship.

Milestone 6: Write the Verdict Document

This milestone produces the real deliverable. Working code shows you can build both designs. The document shows you know when to use each, which is the skill that is hard to fake.

Create VERDICT.md that answers four questions directly, using your own numbers from Milestones 4 and 5:

Did multi-agent decide more accurately? Give the accuracy of each version at each noise level, and name the exact disputes where they disagreed, not only the average.
What did it cost in latency? Give the mean and 95th percentile for each, and say whether the parallel branch recovered any of the extra calls.
What did it cost in tokens and money? Give cost per 1,000 disputes for each version, and multiply by your expected monthly volume so the difference is a real number.
Is the added complexity worth it for this task? State a verdict in one sentence, a recommendation, and one condition under which you would revisit it.

Use this structure for the verdict itself and fill every blank from your CSV files:

TEXT

Verdict: For order-dispute resolution at realistic tool-output sizes,
<single agent | multi-agent> is the better choice.
 
Evidence:
* Accuracy: single <x>% vs multi <y>% at noise 0, <x>% vs <y>% at noise <n>
* Latency (mean / p95): single <a>s / <b>s vs multi <c>s / <d>s
* Cost per 1,000 disputes: single $<e> vs multi $<f>
* Cases where they disagreed: <ids and why>
 
Recommendation: <ship X>. Revisit if <specific condition, for example the
share of disputes needing long seller history grows beyond N%>.

Also include in the repository: the raw CSVs, the three scripts, your original prediction from the start of the project, and one paragraph on what you got wrong. A short note on threats to validity, such as a small test set of 15 cases and one model, makes the document more credible, not less.

Common Mistake
Writing "it depends on the use case" and stopping there. That is true of every design choice and says nothing. The document must state what your data showed for this task, and what you would ship because of it.

Tip
Keep both implementations in the portfolio repo whichever one wins. The decision record, with its numbers and its honest limits, is what shows engineering judgement.

Validation & Testing

Check that both versions work, the data is trustworthy and the document commits to an answer.

Bash

## 1. Both versions return a valid decision for the same dispute. Expected: refund
python -c "from single_agent import run_single; print(run_single('DSP-001')['decision'])"
python -c "from multi_agent import run_multi; print(run_multi('DSP-001')['decision'])"
 
## 2. The harness captures tokens. Expected: no AssertionError and non-zero token columns
RUNS=1 python bench.py
 
## 3. No unparseable answers. Expected: 0
python -c "import pandas as pd; d = pd.read_csv('results-noise0.csv'); print((d.decision == 'unparseable').sum())"
 
## 4. Every case ran for both versions in every repeat. Expected: 30 per version at RUNS=2, 45 at RUNS=3
python -c "import pandas as pd; print(pd.read_csv('results-noise0.csv').groupby('version').size())"

Then verify the human side:

Ground truth: ask someone else to apply the policy table to each of the 15 rows without seeing your expected column, and compare. Any disagreement means the case or the policy is ambiguous, so fix it before trusting an accuracy number.
Fair conditions: confirm both versions use the same MODEL, the same POLICY and the same tools, and that the run order was randomised.
Complete data: the noise 0, 100 and 400 files all exist and have no missing latency or token values.
Committed verdict: reread VERDICT.md and confirm it names one recommendation and one revisit condition, with every number traceable to a CSV.

Question	Single agent	Multi-agent
Accuracy at noise 0, 100, 400
Mean and p95 latency
Cost per 1,000 disputes
Model calls per dispute	about 3	about 7
Coordination overhead	None	Reader, two specialists, decider

Common Mistake
Trusting a result from a single noisy run. Model output varies slightly even at temperature 0 across API conditions, so use at least three repeats per case, and treat differences smaller than that variation as ties.

Found a mistake, an outdated step or a broken link?

Multi-Agent vs Single Agent: Cost and Latency Test

Blueprint Walkthrough

Architecture Overview & Problem Statement

What you build, learn and can show

The task: order dispute resolution

The two designs

Prerequisites

Milestone 1: Set Up the Project and the Fixed Test Set

Milestone 2: Build the Single-Agent Baseline

Milestone 3: Build the Multi-Agent Version

Milestone 4: Measure Both Versions Fairly

Milestone 5: Stress the Baseline with Noisy Data

Milestone 6: Write the Verdict Document

Validation & Testing