Dekita

Your safety check is the outage: cost-aware backoff for governed tool calls

ai agents testing typescript

Your agent gets stuck on a file it could fix in one call. The transcript looks like this:

step 41  read_file config/app.yaml        -> ok
step 42  write_file config/app.yaml       -> BLOCKED by policy: target outside allowlist
step 43  read_file config/app.yaml        -> ok
step 44  write_file config/app.yaml       -> BLOCKED by policy: target outside allowlist
... 60 more steps, same two lines

Nothing errored. The guard did its job on every single call. The run still failed, because the guard was measuring the wrong thing and nobody noticed.

A paper submitted to arXiv on October 6, 2026 puts a name on this: control-generated disturbance. Veronique Ziegler, "When the Governor Becomes the Disturbance: Control-Generated Disturbance and Cost-Aware Backoff in Governed Tool-Using Agents" (arXiv:2610.09037). The short version is that a supervisory layer can become the dominant source of failure in the system it supervises, and that the fix is to feed the supervisor's own interventions back into its own strictness.

The loop, in plain terms

A governor watches tool calls and decides allow or block. When it blocks, the agent sees a failure and tries something else. If the blocked call is the only path to the goal, the agent retries, gets blocked again, and burns the rest of the budget.

The detail that makes this hard to see in production: a blocked call is not a tool failure. It is an intervention. It lands in a different log, a different metric, a different team's dashboard. Your tool error rate stays flat and green while your completion rate drops. The governor is causing the problem and also filtering the evidence of the problem.

The paper is explicit that this matters because of how the cost lands. In its framing, an intervention carries a cost to the agent, and a policy that keeps intervening at a rate the environment cannot absorb turns that cost into persistent action blocking. The governor is not wrong about any individual call. It is wrong about the aggregate.

What the paper actually measured

The setup is a controlled file-recovery environment. The authors raise regulatory intensity, and the increase triggers experimentally imposed tool failures. That is a deliberately artificial cost model, and the paper says so in its own limitations: the cost mechanisms are imposed, and the induced events are directly observable to the backoff rule.

The results, as stated in the abstract:

| Condition | What was run | Reported outcome | |---|---|---| | Hand-coded stochastic-policy agent | Failure pattern under result replacement and under execution of corrupted tool arguments | Blocking pattern appears in both modes | | Persistent policy, adaptive backoff vs fixed weak governor | Intervention frequency approximately matched | Backoff improves completion | | Gemini 2.5 Flash | 576 episodes across 6 tasks | Less blocking, better completion under backoff | | Backoff strength sweep | Several settings | Intermediate strength achieved the highest observed aggregate success |

Read that fourth row twice. Stronger backoff is not better. The best observed aggregate success came from an intermediate setting, which is what you would expect if the governor is doing real work: back off to zero and you lose the protection you built the thing for.

Scale check before you quote this: 576 episodes, 6 tasks, one model, one synthetic environment, single author. That is a signal about a mechanism, not an effect size you should copy into a roadmap. What transfers is the structure of the fix, not the numbers.

Three counters you are probably not keeping

Before any code, fix the instrumentation. If blocked calls are not counted separately from failed calls, you cannot see this class of bug at all.

type CallOutcome =
  | { kind: "ok" }
  | { kind: "tool_error"; message: string }
  | { kind: "blocked"; rule: string }   // your governor said no
  | { kind: "escalated"; reason: string };

Three numbers, all per run:

  1. blocked / attempted. The intervention rate. If this is above a few percent on a task the agent knows how to do, your governor is a participant, not a referee.
  2. blocked_then_blocked. Consecutive blocks on the same logical step. Two or more means the agent has no alternative path left, and more strictness cannot help it.
  3. steps_after_first_terminal_block. How much budget got spent after the run was already unrecoverable. This is the number to put on a chart, because it is pure loss.

The second counter is the one the paper's mechanism predicts. A cost-blind governor keeps applying the same probability of intervention regardless of how many times it has already blocked the same step. The agent's behavior changes; the governor's does not.

The governor, with its own damage as input

The fix is small: keep a moving average of induced events, and lower the intervention probability as that average rises. When the environment calms down, strictness comes back.

type GovernorConfig = {
  baseRate: number;      // intervention probability at rest, 0..1
  window: number;        // observations in the moving average
  strength: number;      // 0 = no backoff, 1 = full backoff
  floor: number;         // never go below this intervention rate
};

class CostAwareGovernor {
  private induced: number[] = [];
  private cfg: GovernorConfig;

  constructor(cfg: GovernorConfig) {
    this.cfg = cfg;
  }

  // 1 when the last intervention was followed by another block on the
  // same step, 0 when it was followed by progress. That distinction is
  // the whole point: it separates "we caught something" from "we caused
  // the loop".
  record(followedByAnotherBlock: boolean): void {
    this.induced.push(followedByAnotherBlock ? 1 : 0);
    if (this.induced.length > this.cfg.window) this.induced.shift();
  }

  private inducedRate(): number {
    if (this.induced.length === 0) return 0;
    const sum = this.induced.reduce((a, b) => a + b, 0);
    return sum / this.induced.length;
  }

  // Effective probability of intervening on this call.
  currentRate(): number {
    const backoff = 1 - this.cfg.strength * this.inducedRate();
    return Math.max(this.cfg.floor, this.cfg.baseRate * backoff);
  }

  decide(risky: boolean, rng: () => number = Math.random):
    { allow: true } | { allow: false; rule: string; rate: number } {
    if (!risky) return { allow: true };
    const rate = this.currentRate();
    return rng() < rate
      ? { allow: false, rule: "policy:risky-call", rate }
      : { allow: true };
  }
}

Notes on the choices, because each one is a place people get it wrong:

Reproduce the block storm in ten minutes

A guard you have never seen fail is a guard you cannot trust. Build a stub tool that fails when strictness crosses a threshold, then run both governors against it.

// A tool that only works when the governing rate is low. The failure is
// induced, not natural: that is the paper's setup, and it is the fastest
// way to see the mechanism.
function makeFragileTool(opts: { tolerance: number }) {
  let strictness = 0;
  let calls = 0;

  return {
    setStrictness(v: number) { strictness = v; },
    call(): { ok: boolean } {
      calls += 1;
      const inducedFailure = strictness > opts.tolerance;
      return { ok: !inducedFailure };
    },
    get calls() { return calls; },
  };
}

function run(cfg: GovernorConfig, tolerance: number, budget = 200) {
  const gov = new CostAwareGovernor(cfg);
  const tool = makeFragileTool({ tolerance });
  let lastBlocked = false;

  for (let step = 0; step < budget; step++) {
    tool.setStrictness(gov.currentRate());
    const decision = gov.decide(true);

    if (!decision.allow) {
      // A block is an intervention. The agent retries the same step.
      gov.record(lastBlocked);     // was the previous block followed by another?
      lastBlocked = true;
      continue;
    }

    const res = tool.call();
    gov.record(false);
    if (res.ok) {
      return { completed: true, steps: step + 1, calls: tool.calls };
    }
    lastBlocked = false;
  }
  return { completed: false, steps: budget, calls: tool.calls };
}

Run it three ways and compare completed and calls:

That third case is why the tuning sweep in the paper lands on an intermediate value. It is also the experiment to run before you pick a strength, on your own tools, with your own budget.

What this does not fix

Start here

If you do one thing today, add the fourth outcome to your tool-call log: blocked, with the rule name. Then chart blocked_then_blocked per run. If that number is above zero, you have a feedback loop, and the moving average above is about forty lines of code away.

The general lesson is worth more than the patch. When a control layer sits inside the loop it is trying to control, its output is also input. Any counter it keeps about the system that excludes its own effect is measuring a system that does not exist.

Sources:

This post was written with AI assistance. The author is responsible for its content.

← All writing