Dekita

Temperature 0, same model, two different GPUs: find the word that splits them, then pin it

ai llm devtools ci

You set the temperature to 0, you fix the seed, you run the same GGUF file, and the two GPUs disagree. One says 20. The other says 2000. Neither of them is wrong about its own arithmetic, and each one repeats itself exactly when you run it a second time. The difference is not in the model, and it is not in the prompt. It is in the order in which each card adds its numbers.

That is the shape of the problem, and this week it got a full, careful write-up you should read: a developer on dev.to, xbill, followed one Gemma 4 question across a laptop GPU and a cloud GPU, token by token, down to the single word where the two answers part, and then down to the arithmetic underneath. This is not that article. He explains why. This one gives you the three things you can run on your own two cards tomorrow: a script that finds the exact token where two runs stop agreeing, the one stock setting that makes them agree again, and a gate that stops a CI run from quietly claiming reproducibility it did not actually check.

What temperature 0 actually pins

Greedy decoding removes one source of randomness: the sampler. At temperature 0 the engine takes the highest-scoring candidate and the seed stops mattering, because nothing is sampled. So if two runs of the same build on the same card agree and two runs on different cards do not, the variation is upstream of the sampler. The engine computed two slightly different probability tables, and greedy decoding faithfully picked from two slightly different tables.

The reason the tables differ is that a logit is the end product of millions of multiply-adds, and the order of those additions is a choice the kernel makes, not a fact about the math. Floating-point addition does not associate. Grouping the same numbers into sixteen partial sums and then adding the sixteen gives a different last digit than grouping them into forty and adding the forty. Each digit is a valid floating-point result. Across a stack of layers, the last digits drift, and the answer only changes if the drift lands on a decision that was close.

That last clause is the whole game. Most tokens win by a wide margin, and a last-bit difference does nothing. The answer changes when the model is at a near-tie, and a one-digit change flips which candidate is on top. From that token on, the two runs are answering different questions, because each one now carries a different context. The flip is bimodal: usually identical, and occasionally completely different from one word onward. "It reproduced when I tested it" is weak evidence, for exactly that reason.

Find the word where they part

You do not need to believe the explanation on trust. Run the same prompt on both cards and find the split yourself. The split is the first token where the two runs rank their candidates differently, and everything after it follows from that one.

#!/usr/bin/env bash
# repro_flip.sh A.txt B.txt
# Compare two model outputs token by token. Print the first token where they
# diverge, with a little context on either side. Exit 0 = identical, 1 = they
# split, 2 = missing input.
set -uo pipefail
A="${1:-}"; B="${2:-}"
if [ ! -f "$A" ] || [ ! -f "$B" ]; then
  echo "usage: $0 <run_a.txt> <run_b.txt>" >&2; exit 2
fi

# Flatten each file into whitespace-separated tokens, one per line.
mapfile -t wa < <(tr '[:space:]' '\n' < "$A" | grep -v '^$' || true)
mapfile -t wb < <(tr '[:space:]' '\n' < "$B" | grep -v '^$' || true)
n=${#wa[@]}; [ "${#wb[@]}" -lt "$n" ] && n=${#wb[@]}

for ((i=0; i<n; i++)); do
  if [ "${wa[$i]}" != "${wb[$i]}" ]; then
    s=$((i-5)); [ $s -lt 0 ] && s=0
    e=$((i+6)); [ $e -gt ${#wa[@]} ] && e=${#wa[@]}
    f=$((i-5)); [ $f -lt 0 ] && f=0
    g=$((i+6)); [ $g -gt ${#wb[@]} ] && g=${#wb[@]}
    echo "first split at token $i of $n"
    echo "A: ... ${wa[@]:$s:$((e-s))} ..."
    echo "B: ... ${wb[@]:$f:$((g-f))} ..."
    echo "A token: ${wa[$i]}   B token: ${wb[$i]}"
    exit 1
  fi
done
echo "no split in the first $n tokens (lengths: A=${#wa[@]} B=${#wb[@]})"
exit 0

Feed it two complete outputs of the same prompt, one per card, and it hands you the exact token and a few on each side. In the Gemma 4 case the split sits on a coin flip between two nearly tied words. By the time you read the final line, the two answers look unrelated. Read the split point and the cause becomes obvious: the model was deciding between two close candidates, and the cards landed on opposite sides.

Run each card against itself first. A difference between cards means something only if each card agrees with itself. If card A differs from an earlier run of card A, the cause is not the other card, and you are chasing the wrong thing.

The one setting that stops it

You have three levers, and they are not the same fix.

The stock one, no patch: tell the engine to process one token at a time for the whole prompt. In llama.cpp that is the micro-batch size set to one. The prompt then goes through the same single-token kernels a generated token uses, which split no work across the card's processors and never hand a product to the vendor math library. That removes both of the two causes at once, because both of them live in how the prompt is batched. The measured cost is about 14 to 15 percent more total run time on short prompts, and the prompt-processing speed drops by an order of magnitude while the generation speed barely moves. Long prompts spend a bigger share of each run in the prompt, so the cost grows with prompt length, and that has not been measured here. Use it when two people on two cards have to be able to reproduce the same score, and say so in the report.

The deeper one, a patch: build with the deterministic numerics flag. A pull request in llama.cpp, number 16016, adds a mode that pins the reduction order for the matrix products and skips the vendor library for them, which is the batch-invariance property that the stock setting only approximates for the prompt. It buys you run-to-run and batch-size invariance inside a given backend, and it is the right tool when your divergence comes from concurrent requests or from the prompt cache rather than from the card.

The one you do not need to build yourself: the vendor math library, per its own documentation, only promises bitwise-stable results across runs on GPUs with the same architecture and the same number of processors. Two cards of the same generation with different processor counts are, by that sentence, allowed to disagree. That is not a defect you can file. It is a documented boundary.

The gate that keeps the claim honest

A reproducibility claim is a promise about a setup, and a setup has a fingerprint: the card, the build, and the settings. Report all three with every score, and gate the comparison on them. Two cards that agree on a total can still disagree on which questions they got right, so the gate has to compare the setup record, not just the number.

#!/usr/bin/env bash
# pin_report.sh "card-name" "build-tag" "settings-string"
# Print one line of JSON: the setup fingerprint, so a CI run can compare it
# against the one that produced a previously reported score.
set -euo pipefail
card="${1:-unknown-card}"
build="${2:-unknown-build}"
settings="${3:-}"
sha=""
if command -v sha256sum >/dev/null 2>&1; then
  sha=$(printf '%s' "$settings" | sha256sum | cut -d' ' -f1 | cut -c1-16)
fi
printf '{"card":"%s","build":"%s","settings_sha":"%s"}\n' "$card" "$build" "$sha"

Wire it into the run that produces a score. When a later run wants to call itself reproducible, it runs the same fingerprint and diffs the two lines. Same card, same build, same settings hash, then the numbers are comparable. Any one of the three differs, and the run is a different sample of the model's near-ties, and the score is an average over which way those ties happened to fall on that card.

Two traps to close before you trust the gate. First, the fingerprint must cover the settings that change the arithmetic, the batch and micro-batch sizes, the prompt cache on or off, and the seed, because those are the ones that move the last bits. A settings string that only records the model name does not separate the cases. Second, the model's own scores cannot do this work for you. In the Gemma 4 study, flagging every answer where any token had a narrow lead caught all eighteen answers that changed grade only by flagging one hundred and sixty four of the two hundred and fifty questions. Most answers pass through a near-tie somewhere, and most near-ties change only the wording. You cannot let the model decide which of its own answers are at risk.

Where this stops

This is about arithmetic on fixed hardware, not about a model that is genuinely sampling. If you are running with a temperature above zero, the flips you are seeing are the intended randomness, and pinning the setup will not remove them, because the sampler is still in the loop. The fix here only makes greedy runs comparable across cards and across concurrent load.

And matching two cards is not the same as making either one more accurate. The stock setting makes the cards agree. It does not pick the right card, and no setup in that study won every quantization rung. Each configuration is its own sample, and the total moves by a few questions either way. If you need a number that two people can reproduce, agree on the setup and report it. If you need a number that is right, that is a different problem, and reproducibility will not answer it for you.

The write-up worth reading is the one that followed a single question from the final answer down to the order of the additions. The part that changes your setup is the smaller one: find the split, pin the setup, and gate the claim on the fingerprint.

This post was written with AI assistance. The author is responsible for its content.

Sources

← All writing