Dekita

A Higher Team Score Is Not a Win. Check the Budget First

ai agents devtools testing

A team of agents just beat your single agent by twelve points. You are about to tell your team swarms are worth the spend. Stop. The number that decides that sentence is not the score. It is the token count, and it is usually the thing nobody recorded.

This week a developer on dev.to, p0rt, had eight agents each review a slice of the multi-agent research, and six of the eight carried the same figure back: 17.2x. Five of the six read it as a plain error multiplier, because the preprint they had open still said the independent agents "amplify errors 17.2x." In the published version the number is a trace-level overhead factor, not a failure rate. His catch was not a second opinion; it was a checker that reopened the paper. That is the right instinct, and this post turns the other half of the problem into a check you can paste: was the comparison even on the same budget?

The study that keeps winning the argument

The paper behind that 17.2x is Kim et al., published in Nature Machine Intelligence in July 2026 under the title "Capable language models can outgrow the benefits of collaboration." It is the largest matched-budget comparison I have found, and the one to hold every swarm pitch against.

They held the prompt, the tool access, and the per-system compute budget constant, and changed only the coordination structure and the model. 260 configurations across six benchmarks, five architectures (a single agent plus independent, centralized, decentralized, and hybrid teams), and three model families.

I read the journal page myself; the first three bullets are its own words, the fourth is relayed from the p0rt post and a Chinese digest. I did not read the full text of the six suites, so I am reporting what the paper states, not what I reproduced.

Why your crew demo is lying to you, on purpose

A swarm demo almost never matches the budget. The single agent gets one context and one budget; the team gets four contexts, a coordinator, and the messages between them. The score goes up a little, and the demo is about "how the team worked together."

The study's own numbers say what that gain is worth. At a matched reasoning-token budget, the team's edge is, on average, nothing. Add more tokens and the team usually wins, because more tokens is the only lever that always works. The paper points at the cause: where the big numbers appeared, token usage alone explained most of the variance. The team won by spending, not by structure.

So the question that separates a real finding from a bill is one sentence: which did you benchmark, the structure, or the budget? The study keeps asking it; your pipeline keeps answering "the structure" without checking.

That is a check, not a vibe. You already have the two numbers that answer it. You ran the baseline and the team; both token counts are in your logs. What you are missing is a gate that refuses to report the score delta until the token delta is ruled out.

The gate: prove the runs were on the same budget

The shape is the same as every other gate in this space. Turn "was this a fair fight" into "how far over the baseline token budget did the team run," and make the pipeline answer before the score means anything.

The script below does exactly that on your run records. Nothing installed, no model, no API key, standard library only.

#!/usr/bin/env python3
import json
import sys

REQUIRED = ("name", "score", "tokens")

def die2(msg):
    print("CANNOT COMPARE: " + msg)
    print("A structure comparison needs a single-agent baseline run named 'baseline'.")
    sys.exit(2)

def fmt_tokens(n):
    try:
        return "{:,.0f}".format(float(n))
    except (TypeError, ValueError):
        return str(n)

def fmt_score(x):
    try:
        return "%.3f" % float(x)
    except (TypeError, ValueError):
        return str(x)

def main():
    if len(sys.argv) != 2:
        die2("usage: budget_gate.py runs.json")
    try:
        with open(sys.argv[1], encoding="utf-8") as fh:
            data = json.load(fh)
    except (OSError, json.JSONDecodeError) as exc:
        die2("could not read runs.json (%s)" % exc)

    runs = data.get("runs") or []
    if not runs:
        die2("no 'runs' array found")
    for r in runs:
        if not isinstance(r, dict) or any(k not in r for k in REQUIRED):
            die2("each run needs name, score, and tokens")

    base = [r for r in runs if r["name"] == "baseline"]
    if len(base) != 1:
        die2("exactly one run must be named 'baseline' (the single agent)")
    base = base[0]

    try:
        tolerance = float(data.get("tolerance", 0.10))
    except (TypeError, ValueError):
        die2("'tolerance' must be a number (default 0.10)")
    tolerance = max(0.0, tolerance)
    limit = base["tokens"] * (1.0 + tolerance)

    teams = [r for r in runs if r["name"] != "baseline"]
    if not teams:
        die2("no team runs to compare against the baseline")

    print("baseline tokens: %s" % fmt_tokens(base["tokens"]))
    print("tolerance: +%d%% -> team tokens must be <= %s"
          % (int(round(tolerance * 100)), fmt_tokens(limit)))
    print("-" * 64)

    budget_wins = 0
    for r in teams:
        if r["tokens"] > limit:
            budget_wins += 1
            verdict = "BUDGET WIN, not structure"
            note = ("spent %s tokens. Do not credit the team; you gave it a bigger budget."
                    % fmt_tokens(r["tokens"]))
        else:
            verdict = "matched budget: structure comparison is fair"
            note = ("within tolerance. The score delta may be attributed to structure, cautiously."
                    )
        print("%-12s score %s (base %s)  ->  %s"
              % (r["name"], fmt_score(r["score"]), fmt_score(base["score"]), verdict))
        print("           %s" % note)
        for label, key in (("wall", "wall_s"), ("cost", "cost")):
            if key in r and key in base:
                print("           %-4s %s (base %s)" % (label, r[key], base[key]))
        print()

    if budget_wins:
        print("RESULT: at least one team run was a budget win.")
        print("Re-run the single agent at the team's token budget first.")
        sys.exit(1)

    print("RESULT: every team run stayed within the token tolerance.")
    print("Report the wall-time and cost deltas alongside the score.")
    sys.exit(0)

if __name__ == "__main__":
    main()

runs.json holds an optional tolerance (default 0.10, so a team may spend up to 10% more tokens and still count as matched) and a runs array. Exactly one run is named baseline, the single agent; the rest are team runs.

Exit code 0: every team run stayed within the token tolerance, so the structure is comparable. Exit code 1: a team run spent more than the tolerance above the baseline, so the delta is a budget delta and must not be credited to the team. Exit code 2: an input problem, no baseline or a missing field, and the comparison is refused rather than guessed at.

A real run I just executed, on a worked example with round numbers, not a production trace. Two team runs, one within budget and one five times over:

baseline tokens: 1,000
tolerance: +10% -> team tokens must be <= 1,100
----------------------------------------------------------------
team-4       score 0.580 (base 0.500)  ->  matched budget: structure comparison is fair
           within tolerance. The score delta may be attributed to structure, cautiously.
           wall 95 (base 42)
           cost 0.07 (base 0.03)

team-9       score 0.520 (base 0.500)  ->  matched budget: structure comparison is fair
           within tolerance. The score delta may be attributed to structure, cautiously.
           wall 210 (base 42)
           cost 0.11 (base 0.03)

RESULT: every team run stayed within the token tolerance.
Report the wall-time and cost deltas alongside the score.

Same baseline, one team run at five times the budget:

team-4       score 0.620 (base 0.500)  ->  BUDGET WIN, not structure
           spent 5,200 tokens. Do not credit the team; you gave it a bigger budget.
           wall 210 (base 42)
           cost 0.41 (base 0.03)

RESULT: at least one team run was a budget win.
Re-run the single agent at the team's token budget first.

That is the whole point. The team score is higher, and the run is still refused. The higher score is the price of five times the tokens. The gate does not need a model to say that.

What "worth the swarm" actually requires

The study's biggest wins, and the ones others reported in the same week, cluster around two conditions, worth keeping as the only two yes questions.

  1. Is there an automatic check on the result? A test suite, a verifier, an oracle. The big coding-swarm numbers come from runs where a grader that is not an agent has the last word. Where nothing can say no, you are mostly paying for one model agreeing with itself several times.
  2. Does the work split into parts that do not depend on each other? The +80.8% case is financial research that fans out into independent sources. The -70.0% case is sequential planning where each step changes the state the next reads. Same team, opposite sign, because of the task shape.

Two yeses, and a larger swarm is worth a pilot at a matched budget. Without two yeses, the evidence stops at a handful of agents; the paper's grid only measures teams of three or four. If the first answer is no, build the check before you build the swarm. That is the order.

What this post might get wrong

The gate is not a verdict that swarms are bad. It is a refusal to let a score carry a claim the tokens do not support. Put it where your run results land, and let it exit 1 before your standup gets a sentence it has to un-say.

Sources

This post was written with AI assistance. The author is responsible for its content.