Replay with egress blocked, or it is not a replay

2026-10-29 ai agents testing devtools

Last week's English dev feeds had the same idea from four directions: an agent broke in production, you open the trace, you replay it, and the bug does not come back. One post (Kruglov, October 25) put a record/replay decorator at the tool boundary. Another (apprs) argued you need an environment manifest before you compare two traces. A third (codehub) turned a session into a replay packet a reviewer can re-execute. A fourth (abd_alrahmanqaroot) replayed 10,851 historical shell commands against a rewritten guard; by that author's own count, the rewrite blocked the 15 that should have been blocked and nothing else.

All four assume the same thing: that when you say "replay", the tool calls are served from the trace and not from the network. Most harnesses do not enforce that. They swap in a cached response when the key matches, and fall through to the real function when it does not. That fallback is silent. The run finishes, the output looks plausible, and you cannot tell from the outside whether you just replayed a recording or rolled the dice again on a slightly different world.

So test it. The check is one command and one comparison:

# Replay with no route to the outside, then look at what changed.
unshare -rn ./replay.sh traces/run-2026-10-27T1420.jsonl

If the process cannot reach the network at all and the original failure still reproduces, the trace is complete enough to debug from. If the failure evaporates, your trace was missing something, and every conclusion you drew from that replay was drawn from a run that quietly re-fetched live data.

The distinction that matters: a replay mode is a setting your harness reports about itself. Kruglov's strict mode is the right default, and it still asks you to trust the same process you are testing, because the fallback to live I/O is code you or your framework wrote. Blocking egress at the OS level is a check performed from outside that process. One is a promise. The other is a measurement. When they disagree, believe the measurement.

Why a diff cannot catch this

A step-by-step diff of two replays compares what the model produced. It does not compare where the inputs came from. Four different causes produce identical-looking diffs:

| Cause | Where it lives | Caught by an offline replay? | | --- | --- | --- | | Model behavior changed | provider side | Only if the sampled response was recorded | | Tool implementation or schema changed | your repo | Yes, as a missing cache entry | | Prompt template or dependency changed | your repo | Yes, cheaply | | Live data moved under you | the outside world | Yes, this is the whole point |

Only the first row is a model problem. The other three are yours, and an egress-blocked replay separates them in one run because the live world is not available to paper over a gap.

There is a second reason to care. Anthropic put OpenTelemetry trace export for the Claude Platform into public beta on October 14, 2026, covering Managed Agents, the Messages API, and the SDKs. That is a good development and it does not solve this problem. Platform spans record what the API and the tools did, and on the Managed Agents path the model's reasoning content is not included at all, according to an analyst write-up of the launch. The docs also warn that span names, attributes, and SDK options can change before general availability. An observability backend is where you read a trace. It is not a substitute for having the response bytes on disk in a form your harness can serve without asking anyone.

The five fields that make offline replay possible

Recording "the tool result" is not enough. These five fields are what the blocked run needs:

The raw response body, before your parser touched it. If your recorder serializes a dataclass after validation, you have thrown away the ability to test the parser, which is where a large share of tool bugs live.
The tool schema hash at call time. A replayed call whose arguments no longer validate against the current schema is a real finding, not noise, but only if you recorded which schema was in force when the original call succeeded.
The model identifier that actually answered, read from the response metadata rather than from your config. An endpoint that serves several models behind one name makes the configured value a wish, not a record.
A stable time source. Any tool that reads the clock, and any rate limiter that counts windows, will behave differently on replay day. Inject a fixed clock in replay mode or the comparison is theater.
A failure signature: a hash over the step index, the tool name, the error class, and the exit code. This is what you compare; comparing full transcripts by eye does not scale and comparing them byte for byte fails on every timestamp.

Fields 1 and 3 are the ones most often skipped, and they are the two that decide whether a replay is a replay.

What the blocked run tells you

Three outcomes, three different meanings. Do not collapse them into pass or fail.

The same failure reproduces. The trace is sufficient for this bug. Go debug the recorded data. Note the limitation: a reproduction proves the trace contains the necessary inputs, not that the agent's reasoning was sound.

Nothing fails, or a different failure appears. The trace is incomplete. Something the run depended on was never written down. The usual culprits are a tool response that was only partially persisted, a clock read, or a payload fetched through a path your recorder does not wrap. Fixing the recorder is the work; do not "fix" this by re-enabling the network, which is how the original false confidence got built.

The harness refuses to start. This is the healthiest of the three. It means strict mode is on and an unrecorded call is a hard error rather than a silent live fetch. Strict mode is worth the annoyance precisely because it converts an invisible re-run into a visible missing entry.

A blocked run that passes is also a cheap regression test. Keep it in CI: any change that adds a new outbound dependency to the agent path will fail the blocked replay before it reaches production, which is earlier and cheaper than finding out during an incident.

Making the check portable

unshare is Linux-only, and it needs user namespaces enabled. On macOS or on a locked-down CI runner you will not have it. A fallback that works anywhere: wrap your HTTP client so every outbound destination is appended to a per-run log, then assert that the replay-mode log is empty.

# egress.py: run this around the replay, not around the agent's own code.
import json, os

def assert_offline(log_path: str) -> dict:
    dests = set()
    if os.path.exists(log_path):
        with open(log_path, encoding="utf-8") as f:
            for line in f:
                line = line.strip()
                if line:
                    dests.add(json.loads(line)["dest"])
    return {"outbound_destinations": sorted(dests), "clean": len(dests) == 0}

If that comes back dirty, print the destination set before the trace. It names the exact dependency your recorder is missing, which is more useful than a pass or fail.

One honest limit: a client wrapper only sees traffic that goes through it. A subprocess, a native extension, or a tool binary that opens its own socket will escape. That is why the namespace check is the stronger version and the wrapper is the convenient one. Use the namespace where you can, and treat a clean wrapper log as weaker evidence, not proof.

What I am not claiming

I have not measured how often traces are incomplete. The check tells you about your own run, and that is the only claim it supports.
Offline replay does not prove the original run was correct. It proves it is reproducible. A faithfully replayed wrong answer is still wrong.
Recording raw bodies has a storage and PII cost. Truncate deliberately and record the truncation, or you will replay a response that was never the response.

The engineering claim is narrow and worth stating plainly: if a replay needs the network, you are not debugging the run that failed. You are running a new one and calling it evidence. Block egress for one run today. The result takes seconds, and it is the only cheap way to find out which of those two things your harness has been doing.

Sources: Kruglov, "Replay or It Didn't Happen: Making Agent Runs Reproducible" (DEV Community, October 25, 2026); apprs, "The Diff Wasn't the Model: Fingerprint the Environment Before You Replay a Trace" (DEV Community, October 26, 2026); codehub, "The Replay Packet: Turning an Agent Session Into Evidence a Reviewer Can Re-run" (DEV Community); abd_alrahmanqaroot, "Replay your history before you trust an agent guardrail" (DEV Community); Anthropic Claude Platform OpenTelemetry trace export, public beta announced October 14, 2026; Futurum Group, "Claude Agent Observability Can't Be the Audit Record on Its Own" (analysis of that launch). Two of the DEV Community dates were not shown on the pages I read, so they are cited without a date rather than with a guessed one.

This post was written with AI assistance. The author is responsible for its content.