Dekita

A yes vote that does not say what it read is not evidence

ai agents devtools architecture

Two agents agreed. Both were wrong. In a twelve-day run of a local orchestrator for coding agents, a Staff agent decided the fix for a broken dataset feature was to add a new API to the project's Python engine, so the user's code would run. A researcher supported the plan. The user's code was AI-generated and simply wrong. The engine never had that API. The author closed the job and wrote up the failure in one line: "Two agents agreed and neither checked the premise."

The same run produced a quieter problem. Researchers sometimes supported a plan without reading the code. They said so in the reply itself, in a form like "Checked board msg1625 only; no code, PR/diff/branch, tests or CI inspected." On the message board, that support looks identical to support backed by the code. The author's own list of what to fix next names both failures: let agents say honestly when something was not checked, and make them show what they based their answer on.

The decision record below answers both. One JSONL file, one script, four rules: every vote lists what the voter actually read, unverified votes cannot bind, decisions are immutable and changed by superseding rather than editing, and a decision that changes code names the premise check that ran before it bound.

The run, and what the numbers do and do not say

The material comes from two posts by Adam Drag about Triglav, a local-first orchestrator that runs Claude Code and Codex agents like a small software consultancy, published on dev.to on October 17 and October 19, 2026.

From the October 19 follow-up, covering October 7 to 19: 1,648 messages on 76 jobs. 479 of the messages were replies, and 42 of the replies were challenges. 192 messages were routed to the author, who made 176 decisions. There were 311 blockers, and 112 of them only said that the Staff agent was offline. The author's rough estimate is that 15 to 25 percent of the messages were real discussion.

From the October 17 release post: in its first ten days the setup ran 41 jobs on the author's own repositories and finished 34 of them. The review panel finished 96 rounds, 54 passed and 42 asked for changes. In the first six days, all agents together wrote about 3.6 million output tokens and read about 3.6 billion from the prompt cache, roughly a thousand to one. A long-running Staff session often carries 300k to 700k tokens of context.

Every one of those numbers is self-reported by one person, on one person's repositories, with one tool. Triglav itself is source-available under the Business Source License 1.1, free for your own and your company's internal work, not open source, with each version moving to GPL-2.0-or-later after four years. Treat the figures as one honest data point, not a benchmark.

Three ways a message board lies

Support without reading. A vote that says "support" and lists nothing it read is not evidence, it is a mood. The board cannot tell the two apart unless the vote carries its own reading list.

Accepts that arrive too fast. Staff accepted almost every challenge, and often that was the right call. But in some threads the accept came about a minute after the challenge, and the author cannot tell whether anything was checked. A decision that changes one minute after the objection it answers deserves the same suspicion as one that never answers it. The record does not make accepts slower. It makes them measurable: every decision carries a timestamp next to the challenge it answers, so a pattern of one-minute accepts shows up in the log instead of in somebody's memory.

Tools that report things that are not true. One tool field only accepted "CI: success", even for jobs with no CI at all. One Senior refused to call the field at all, because the value would have been false. The author marks that one as his own bug. When the schema has no way to say "not applicable", agents either lie or stall.

The record

One JSONL file, append-only. Five record types:

{"id":"d1","type":"question","job":"dataset-feature","question":"Add a new API to the engine so the user's script runs?","context":["user script calls engine.load_dataset(path)","no load_dataset seen under src/engine"]}
{"id":"d2","type":"vote","qid":"d1","agent":"researcher-1","stance":"support_unverified","reason":"the user's code is the spec","read":[],"verified":false}
{"id":"d3","type":"vote","qid":"d1","agent":"staff","stance":"inform","reason":"the premise needs checking first","read":["src/engine/__init__.py"],"verified":true}
{"id":"d4","type":"check","qid":"d1","command":"rg -c 'def load_dataset' src/engine/","exit_code":1,"note":"0 matches: the API does not exist"}
{"id":"d5","type":"decision","qid":"d1","decider":"staff","decision":"reject: the premise is false, the user's code is wrong","check":"d4","supersedes":null}

The rules the format enforces:

  1. A vote stores the list of paths, tickets or messages the voter actually read. support with an empty list is written as support_unverified. Unverified votes can inform; they cannot bind.
  2. A decision binds when at least one verified vote backs it, support or inform, or when the decider lists what it read itself. Otherwise the bind is refused.
  3. A decision that changes code names the check that ran before it bound, with the check's exit code. No check, no bind.
  4. Decisions are immutable. A change appends a new decision record with supersedes pointing at the old id. The old record stays in the file.

The script

Node 18 or newer, no dependencies. The demo subcommand replays the failed dataset job above and shows both refusals.

#!/usr/bin/env node
/**
 * decision-record.js - a vote that does not say what it read is not evidence.
 *
 * Append-only JSONL log for multi-agent decisions. Enforced rules:
 *   R1  Every vote lists what the voter actually read. Support with an
 *       empty read list is stored as support_unverified and cannot bind.
 *   R2  A decision binds on a verified vote that backs it (support or
 *       inform), or on the decider's own read list. Otherwise the bind
 *       is refused.
 *   R3  A decision that changes code names the premise check that ran
 *       first, with its exit code. No check, no bind.
 *   R4  Decisions are immutable; a change supersedes, it never edits.
 *
 *   node decision-record.js log.jsonl demo
 *   node decision-record.js log.jsonl show
 */
"use strict";

const fs = require("fs");

function load(file) {
  if (!fs.existsSync(file)) return [];
  return fs.readFileSync(file, "utf8").split("\n")
    .filter((l) => l.trim())
    .map((l) => JSON.parse(l));
}

function append(file, rec) {
  const log = load(file);
  rec.id = "d" + (log.length + 1);
  rec.ts = new Date().toISOString();
  fs.appendFileSync(file, JSON.stringify(rec) + "\n");
  return rec;
}

function ask(file, job, question, context) {
  return append(file, { type: "question", job, question, context });
}

function vote(file, qid, agent, stance, reason, read) {
  const list = Array.isArray(read) ? read : [];
  const verified = list.length > 0;
  const stored = stance === "support" && !verified ? "support_unverified" : stance;
  return append(file, {
    type: "vote", qid, agent, stance: stored, reason,
    read: list, verified,
  });
}

function check(file, qid, command, exitCode, note) {
  return append(file, { type: "check", qid, command, exit_code: exitCode, note });
}

function bind(file, qid, decider, decision, opts) {
  const o = opts || {};
  const deciderRead = o.deciderRead || [];
  const touchesCode = o.touchesCode === true;
  const log = load(file);
  const votes = log.filter((r) => r.type === "vote" && r.qid === qid);
  const verifiedSupport = votes.filter(
    (v) => v.verified && (v.stance === "support" || v.stance === "inform")
  );
  if (verifiedSupport.length === 0 && deciderRead.length === 0) {
    return { ok: false, why: "no verified support, and the decider lists nothing it read" };
  }
  if (touchesCode && !o.check) {
    return { ok: false, why: "decision changes code but names no premise check" };
  }
  return {
    ok: true,
    record: append(file, {
      type: "decision", qid, decider, decision,
      check: o.check || null, supersedes: null,
    }),
  };
}

function supersede(file, oldId, decision, reason) {
  const log = load(file);
  const old = log.find((r) => r.id === oldId);
  if (!old || old.type !== "decision") return { ok: false, why: "no such decision" };
  return {
    ok: true,
    record: append(file, {
      type: "decision", qid: old.qid, decider: "human",
      decision, reason, check: old.check, supersedes: oldId,
    }),
  };
}

function show(file) {
  const log = load(file);
  const dead = new Set(log.filter((r) => r.supersedes).map((r) => r.supersedes));
  for (const r of log) {
    const mark = dead.has(r.id) ? "  [superseded]" : "";
    if (r.type === "question") console.log(r.id + " Q  " + r.question);
    if (r.type === "vote") {
      console.log(r.id + " V  " + r.agent + ": " + r.stance + " - " + r.reason +
        " (read: " + (r.read.join(", ") || "nothing") + ")");
    }
    if (r.type === "check") {
      console.log(r.id + " C  " + r.command + " -> exit " + r.exit_code +
        (r.exit_code === 0 ? "" : "  <-- premise failed"));
    }
    if (r.type === "decision") {
      console.log(r.id + " D  " + r.decider + ": " + r.decision +
        (r.supersedes ? " (supersedes " + r.supersedes + ")" : "") + mark);
    }
  }
}

function demo(file) {
  fs.rmSync(file, { force: true });
  const q = ask(file, "dataset-feature",
    "Add a new API to the engine so the user's script runs?",
    ["user script calls engine.load_dataset(path)", "no load_dataset seen under src/engine"]);
  vote(file, q.id, "researcher-1", "support", "the user's code is the spec", []);
  const a = bind(file, q.id, "staff", "add engine.load_dataset",
    { deciderRead: [], touchesCode: true });
  console.log("bind 1: REFUSED - " + a.why);
  vote(file, q.id, "staff", "inform", "the premise needs checking first",
    ["src/engine/__init__.py"]);
  const b = bind(file, q.id, "staff", "add engine.load_dataset",
    { deciderRead: ["src/engine/__init__.py"], touchesCode: true });
  console.log("bind 2: REFUSED - " + b.why);
  const c = check(file, q.id, "rg -c 'def load_dataset' src/engine/", 1,
    "0 matches: the API does not exist");
  const d = bind(file, q.id, "staff", "reject: the premise is false, the user's code is wrong",
    { deciderRead: ["src/engine/__init__.py"], touchesCode: true, check: c.id });
  console.log("bind 3: BOUND as " + d.record.id);
  console.log("");
  show(file);
}

const argv = process.argv.slice(2);
const file = argv[0];
const cmd = argv[1];
if (cmd === "demo") demo(file);
else if (cmd === "show") show(file);
else console.log("usage: node decision-record.js <file> demo|show");

The premise check is the part that catches the wrong agreement

A premise check is a command whose exit code answers the question the agents agreed on. For "does this API exist", it is a search of the source tree. For "is this behavior already implemented", it is a test that fails before the change and passes after. For "does this dependency exist", it is a registry query. The check runs before the bind, and its command and exit code go into the log next to the decision, so a reader can rerun it months later.

This is the step the failed job skipped. Both agents had read the message board. Neither had read the engine. A grep takes a second, and the whole disagreement dissolves. The point is not distrust of the agents. It is that a premise is a claim about the world, and claims about the world are cheap to check, so there is no excuse for binding a decision on one that nobody ran.

What the record does not fix

It does not make the board quieter. If 15 to 25 percent of messages are real discussion, the record makes the discussion searchable, not smaller. It does not stop the 112 blockers that only said Staff was offline; those are a runner problem, not a records problem. It does not touch the token economics, where six days of wake-ups read a thousand cache tokens for every token written; the release post's own answer there was to wake Staff only for events it needs and to let it wait up to 50 minutes per call on its one-hour prompt cache, and records do not change that arithmetic. And it does not prove a decision was right. It proves who decided what, on what evidence, and when it changed, which is the part you need when the next wrong premise arrives.

Sources

This post was written with AI assistance. The author is responsible for its content.

← All writing