Dekita

You are not going to read the code. Read the harness instead.

ai agents testing ci

When an agent-authored pull request claims it fixed something, reverse the fix and run the suite. If everything stays green, the suite is not testing the change. One script, nothing but git:

#!/usr/bin/env bash
# revert_test.sh - answer one question about a change under review:
#   "If this change were removed, would the test suite notice?"
# A suite that stays green after the fix is reversed does not test the fix.
#
# Usage: ./revert_test.sh <commit-ish> [test command]
#   Default test command: npm test
# Exit codes:
#   0 = the suite saw the change (at least one test failed after the revert)
#   1 = the suite stayed green (the change is invisible to the tests)
#   2 = the check could not run (bad args, git failure, binary files)

set -u

COMMIT="${1:-}"
TESTCMD="${2:-npm test}"

if [ -z "$COMMIT" ] || [ -z "$TESTCMD" ]; then
  echo "usage: $0 <commit-ish> [test command]" >&2
  exit 2
fi

if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "not a git repository" >&2
  exit 2
fi

WT="$(mktemp -d)"
trap 'git worktree remove --force "$WT" 2>/dev/null' EXIT

git worktree add --detach "$WT" HEAD >/dev/null 2>&1 || {
  echo "could not create a worktree" >&2
  exit 2
}

if ! git -C "$WT" show "$COMMIT" >/dev/null 2>&1; then
  echo "cannot read commit: $COMMIT" >&2
  exit 2
fi

# Reverse-apply the change, leaving it reversed in the worktree only.
git -C "$WT" show "$COMMIT" | git -C "$WT" apply -R || {
  echo "reverse-apply failed; the change may touch binary files" >&2
  exit 2
}

# Run the suite with the fix removed.
if $TESTCMD; then
  echo "GREEN after revert: the suite does not test this change."
  exit 1
else
  echo "RED after revert: the suite sees the change."
  exit 0
fi

That is the whole trick. The rest of this post is why it matters now, and the three other checks that do the job that reading the diff line by line used to pretend to do.

The review that stopped scaling

In mid-October, Anthropic disclosed that Claude writes about 90 percent of Claude Code's own code. Between June and September, merged pull requests to that codebase more than doubled, while the share that had to be reverted fell by more than 40 percent. Those are Anthropic's own figures, and I am reading them secondhand, through a dev.to writeup of the team's blog post and Chinese tech press coverage of the same post. I have not read the original.

The framing matters more than the numbers. As relayed, the team treats "did a human read the diff" as a rough proxy. The question they ask instead is what could go wrong with this change, and which check would catch it. That is the reframe of someone whose arithmetic stopped working, not someone who stopped caring.

An October 21 dev.to post put the collapse plainly: nobody was reading the code anyway. On a large service, "I reviewed it" meant "I sampled it and I trusted the sampler." Agents did not invent sampling. They changed the ratio. The volume of code went up by an order of magnitude and the reading stayed flat, so the sample rate collapsed.

If you accept that, the useful question is not how to read faster. It is what to read instead.

Four things to read instead of the code

  1. Whether the test suite sees the change under review.
  2. Whether the eval set contains the case that prompted the change.
  3. What capability the diff adds, not how the logic is written.
  4. The harness that judges the output, because the harness is code too.

1. The revert test

The script above is the first check. The routine from the October 21 post: revert the fix, run the test, watch it go red. If it stays green, the test is decoration. That post's author reports catching more fake tests this way than real bugs, which is one practitioner's base rate, not a measured frequency.

The judgment call is in the exit code. Green after the revert means the suite does not see this change. That is either a missing test or a decorated one, and the script does not tell you which. Both are worth a conversation before merge, and neither is visible from reading the diff.

Two limits worth knowing. The check needs a runnable suite; a repo whose tests need an hour and three services will not get this run on every PR. And for a pull request that adds new behavior, green after the revert means the new behavior has no test that fails without it. That is a coverage gap to close, not proof of a decorated test.

2. The eval that remembers

Second check, one question: is the bug that prompted this pull request in the eval set? If it is not, the PR fixed one instance and taught the system nothing. The eval is the memory. Without it you re-fix the same bug with different variable names every few weeks, and each fix looks like progress.

This is the cheapest item on the list and the one most often skipped, because it is not a script. It is a filing habit: every bug report gets a failing case in the eval before it gets a fix in the code.

3. Blast radius over logic

Third check, and the one reviewers already do informally: stop asking how the function is written and ask what the diff added. A network call, a new dependency, an environment variable, a file write outside the sandbox. Agents are good at adding a dependency that duplicates one you already have, and bad at mentioning it.

The added lines of a diff are a short list. Scan them for capability changes:

# One-liner, not a script: replace <commit-ish> with the commit under review.
# Only the added lines, only the patterns that change what the code can do.
# The environment-access pattern covers the common spellings across runtimes.
git show <commit-ish> | grep '^+' | grep -nE 'process\.e[n]v|os\.environ|os\.getenv|Getenv|subprocess|os\.system|fetch\(|requests\.|urllib|socket|"dependencies"|open\('

The pattern list is a starting point, not a standard. Extend it with whatever your runtime can reach: database drivers, message queues, credential loaders. The point is that the question "what can this do now that it could not do before" is answerable from the diff in a way that "is this logic correct" is not.

4. The harness is also code

Fourth check, the uncomfortable one. When the artifact under review is generated, the thing that judges it is the harness: the eval set, the fixtures, the assertions, the scorer. Someone wrote the scorer. If the scorer was agent-written and unread, you have built a mirror, checking generated output against generated judgment and calling the agreement a pass.

So read the scorer by hand. Always. It is usually the smallest file in the repo and it decides everything. Same for the fixture loader. Everything else can be agent-written and skimmed.

The reviewer needs a stop condition too

The same week produced the same problem inside one team, from the other side. As relayed by a secondary writeup, one pull request to a command-line tool cycled through more than 50 rounds of review overnight, with an agent author and an agent reviewer arguing, and 59 percent of repeated findings were the reviewer re-raising its original pass rather than surfacing new bugs. That figure comes from a single relay, so treat it as direction rather than measurement.

The reported remedies: after three rounds the reviewer must ask the author to reconsider the approach instead of commenting again, later passes review only the new diff, and no finding may be posted twice. The reported results: pull requests passing after a single review-and-fix cycle rose from 65 to 78 percent, and those needing five or more rounds fell from 8.9 to 1.5 percent.

The transferable lesson is not the numbers. It is that a review agent is a model with the same failure modes as the author, so it needs the two things every long-running loop needs: a stop condition, and an eval set built from real incidents and reverts, scored on every change to the reviewer itself.

The unglamorous telemetry

One more thing accuracy evals cannot see: silent exception swallowing, a retry loop that turns one failed request into a bill, a test that passes because it asserts nothing, a scorer that returns full marks for an empty string. None of that shows up in a benchmark that measures whether the output was correct.

So log the boring numbers per run: cost, wall clock, outbound call count, diff size over time. The October 21 post's author reports that this log has caught more regressions than any eval they wrote. That is a single practitioner's experience, not a study, but it is the kind of claim you can check on your own repo in an afternoon.

What this does not cover

Sources

This post was written with AI assistance. The author is responsible for its content.

← All writing